Jul 01 16:56:10.162458 ip-10-0-141-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 16:56:10.162472 ip-10-0-141-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 16:56:10.162481 ip-10-0-141-111 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 16:56:10.162776 ip-10-0-141-111 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 16:56:20.212073 ip-10-0-141-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 16:56:20.212093 ip-10-0-141-111 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c654acebfff34e129452f587f29024be --
Jul 01 16:58:28.517341 ip-10-0-141-111 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 16:58:28.885560 ip-10-0-141-111 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 16:58:28.885560 ip-10-0-141-111 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 16:58:28.885560 ip-10-0-141-111 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 16:58:28.885560 ip-10-0-141-111 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 16:58:28.885560 ip-10-0-141-111 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 16:58:28.887813 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.887721    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 16:58:28.893423 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893400    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:28.893423 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893418    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:28.893423 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893421    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:28.893423 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893424    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:28.893423 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893427    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:28.893423 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893431    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893434    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893437    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893439    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893443    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893446    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893453    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893456    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893459    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893461    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893464    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893467    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893469    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893472    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893475    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893478    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893480    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893483    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893486    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893488    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:28.893636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893491    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893494    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893497    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893499    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893502    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893504    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893507    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893509    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893512    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893514    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893517    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893519    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893522    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893527    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893531    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893534    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893536    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893540    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893542    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893545    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:28.894149 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893548    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893550    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893553    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893556    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893558    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893561    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893564    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893566    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893569    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893571    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893574    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893576    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893580    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893582    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893586    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893588    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893591    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893593    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893596    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:28.894636 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893600    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893603    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893606    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893609    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893612    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893616    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893619    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893622    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893626    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893628    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893631    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893633    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893636    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893639    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893641    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893644    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893648    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893650    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893653    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893656    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:28.895099 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893659    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893661    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893663    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893666    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893669    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893672    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893674    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893677    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893679    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893683    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893686    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893689    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893692    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893694    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.893697    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894450    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894457    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894461    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894464    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:28.895577 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894467    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894472    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894476    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894479    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894482    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894485    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894488    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894492    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894496    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894511    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894515    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894518    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894521    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894524    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894527    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894530    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894533    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894536    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894539    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:28.896022 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894541    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894544    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894547    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894549    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894552    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894554    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894557    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894559    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894562    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894565    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894568    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894571    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894574    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894577    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894579    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894583    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894586    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894589    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894591    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:28.896519 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894594    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894596    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894599    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894601    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894605    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894607    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894610    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894612    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894615    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894617    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894620    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894622    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894625    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894627    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894630    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894633    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894635    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894638    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894641    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894643    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:28.896987 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894646    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894649    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894652    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894655    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894657    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894660    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894663    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894665    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894668    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894671    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894674    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894676    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894679    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894681    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894684    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894686    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894689    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894692    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894694    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894697    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:28.897480 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894699    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894702    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894704    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894707    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894709    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894712    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894715    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894717    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894720    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894722    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894725    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894728    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894730    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894733    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894737    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894740    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.894743    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895126    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895132    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:28.897951 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895135    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895139    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895144    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895147    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895150    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895153    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895156    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895159    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895162    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895165    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895167    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895170    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895173    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895175    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895178    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895181    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895183    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895186    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895188    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895191    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:28.898419 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895194    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895196    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895199    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895201    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895204    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895207    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895212    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895215    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895217    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895220    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895223    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895226    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895228    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895231    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895234    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895237    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895240    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895242    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895245    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895247    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:28.898910 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895250    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895254    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895257    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895260    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895263    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895266    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895269    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895271    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895274    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895276    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895279    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895281    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895284    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895286    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895289    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895291    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895294    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895296    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895300    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895303    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:28.899413 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895306    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895308    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895313    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895316    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895319    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895321    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895324    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895326    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895329    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895332    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895334    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895337    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895340    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895342    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895345    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895349    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895353    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895356    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895358    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:28.899917 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895361    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895363    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895366    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895368    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895371    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895373    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895376    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895378    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895381    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895383    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895386    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895390    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895393    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895395    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895398    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895402    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895404    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.895407    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895475    2556 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895487    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 16:58:28.900385 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895492    2556 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895496    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895501    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895505    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895509    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895514    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895517    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895520    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895523    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895526    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895530    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895533    2556 flags.go:64] FLAG: --cgroup-root=""
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895535    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895538    2556 flags.go:64] FLAG: --client-ca-file=""
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895541    2556 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895544    2556 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895549    2556 flags.go:64] FLAG: --cluster-domain=""
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895551    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895554    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895558    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895562    2556 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895565    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895568    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 16:58:28.900857 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895572    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895576    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895579    2556 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895582    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895587    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895590    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895593    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895598    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895601    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895604    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895606    2556 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895609    2556 flags.go:64] FLAG: --enable-server="true"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895613    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895617    2556 flags.go:64] FLAG: --event-burst="100"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895620    2556 flags.go:64] FLAG: --event-qps="50"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895623    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895626    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895629    2556 flags.go:64] FLAG: --eviction-hard=""
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895633    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895636    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895639    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895641    2556 flags.go:64] FLAG: --eviction-soft=""
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895644    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895647    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895650    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 16:58:28.901415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895653    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895655    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895658    2556 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895661    2556 flags.go:64] FLAG: --feature-gates=""
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895667    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895670    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895673    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895676    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895679    2556 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895682    2556 flags.go:64] FLAG: --help="false"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895685    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-141-111.ec2.internal"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895689    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895692    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895695    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895698    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895701    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895704    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895707    2556 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895709    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895712    2556 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895715    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895718    2556 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895721    2556 flags.go:64] FLAG: --kube-reserved=""
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895724    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 16:58:28.902092 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895727    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895730    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895732    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895735    2556 flags.go:64] FLAG: --lock-file=""
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895738    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895740    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895743    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895749    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895752    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895754    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895757    2556 flags.go:64] FLAG: --logging-format="text"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895760    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895763    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895766    2556 flags.go:64] FLAG: --manifest-url=""
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895769    2556 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895774    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895777    2556 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895781    2556 flags.go:64] FLAG: --max-pods="110"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895784    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895787    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895790    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895793    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895796    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895804    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895807    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 16:58:28.902681 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895814    2556 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895817    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895820    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895824    2556 flags.go:64] FLAG: --pod-cidr=""
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895827    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895833    2556 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895836    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895839    2556 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895841    2556 flags.go:64] FLAG: --port="10250"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895844    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895847    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0282d08cb77b7e8b1"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895851    2556 flags.go:64] FLAG: --qos-reserved=""
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895854    2556 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895857    2556 flags.go:64] FLAG: --register-node="true"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895859    2556 flags.go:64] FLAG: --register-with-taints=""
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895863    2556 flags.go:64] FLAG: --registry-burst="10"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895866    2556 flags.go:64] FLAG: --registry-qps="5"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895869    2556 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895872    2556 flags.go:64] FLAG: --reserved-memory=""
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895875    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895878    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895881    2556 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895884    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895887    2556 flags.go:64] FLAG: --runonce="false"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895890    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 16:58:28.903293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895893    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895896    2556 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895899    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895902    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895905    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895908    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895911    2556 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895914    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895917    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895919    2556 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895922    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895925    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895928    2556 flags.go:64] FLAG: --system-cgroups=""
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895931    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895936    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895939    2556 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895942    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895947    2556 flags.go:64] FLAG: --tls-min-version=""
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895950    2556 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895952    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895955    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895958    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895960    2556 flags.go:64] FLAG: --v="2"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895964    2556 flags.go:64] FLAG: --version="false"
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895968    2556 flags.go:64] FLAG: --vmodule=""
Jul 01 16:58:28.903893 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895972    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.895975    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896106    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896111    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896114    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896117    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896120    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896122    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896125    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896128    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896131    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896134    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896136    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896139    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896141    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896144    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896146    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896149    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896152    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896154    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:28.904513 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896157    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896160    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896164    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896168    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896172    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896175    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896178    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896181    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896184    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896186    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896189    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896191    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896194    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896197    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896199    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896202    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896204    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896207    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896209    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896212    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:28.905189 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896216    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896219    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896222    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896224    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896227    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896229    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896232    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896235    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896237    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896240    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896242    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896245    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896247    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896250    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896256    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896259    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896262    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896264    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896268    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:28.905973 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896271    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896274    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896277    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896280    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896282    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896285    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896288    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896290    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896293    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896295    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896298    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896300    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896303    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896307    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896310    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896312    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896315    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896318    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896320    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896323    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:28.906459 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896325    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896328    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896330    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896333    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896335    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896338    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896340    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896345    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896347    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896350    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896353    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896355    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896358    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896360    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896363    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896366    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896369    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896373    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896376    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:28.906953 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896378    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896381    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.896384    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.896389    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.903286    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.903301    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.904184    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.904192    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904249    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904254    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904259    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904262    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904266    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904269    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:28.907475 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904272    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904275    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904279    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904283    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904286    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904290    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904293    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904296    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904299    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904302    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904305    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904308    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904310    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904313    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904316    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904318    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904321    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904324    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904326    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904329    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:28.907831 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904331    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904334    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904337    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904342    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904345    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904347    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904350    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904352    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904355    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904357    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904361    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904363    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904366    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904368    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904371    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904373    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904376    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904379    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904381    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904384    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:28.908341 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904386    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904389    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904391    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904394    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904396    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904399    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904402    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904405    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904407    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904410    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904412    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904415    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904418    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904421    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904424    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904426    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904430    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904432    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904435    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:28.908830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904437    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904440    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904443    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904445    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904448    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904450    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904453    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904457    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904461    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904464    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904466    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904469    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904472    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904475    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904477    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904480    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904483    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904485    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904488    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904491    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:28.909382 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904494    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904496    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904498    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904501    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904504    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904506    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904510    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904513    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904517    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904520    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904524    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904527    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904530    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904532    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.904538    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 16:58:28.909855 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904775    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904788    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904793    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904798    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.904802    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905574    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905624    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905633    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905639    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905644    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905649    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905654    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905659    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905664    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905668    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905673    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905678    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905689    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905694    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905699    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 16:58:28.910237 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905703    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905708    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905713    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905717    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905722    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905727    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905762    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905778    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905783    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905788    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905794    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905800    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905805    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905987    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.905999    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906002    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906005    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906008    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906011    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906015    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 16:58:28.911018 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906018    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906021    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906024    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906027    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906030    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906047    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906068    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906080    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906083    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906086    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906089    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906092    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906095    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906098    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906104    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906106    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906109    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906112    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906114    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906117    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 16:58:28.911783 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906120    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906124    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906127    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906129    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906132    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906135    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906138    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906141    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906144    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906146    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906149    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906152    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906154    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906159    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906167    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906173    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906179    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906182    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 16:58:28.912299 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906188    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906192    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906195    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906198    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906201    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906204    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906206    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906209    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906212    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906214    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906217    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906221    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906223    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906226    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906229    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906231    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906234    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906237    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906240    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906242    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 16:58:28.912834 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:28.906245    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 16:58:28.913536 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.906251    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 16:58:28.913536 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.906383    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 16:58:28.913536 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.908773    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 16:58:28.913536 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.910238    2556 server.go:1019] "Starting client certificate rotation"
Jul 01 16:58:28.913536 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.910355    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 16:58:28.913536 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.910395    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 16:58:28.913536 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.912652    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 16:58:28.913831 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.913816    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 16:58:28.920714 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.920700    2556 log.go:25] "Validated CRI v1 runtime API"
Jul 01 16:58:28.923276 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.923263    2556 log.go:25] "Validated CRI v1 image API"
Jul 01 16:58:28.924027 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.924005    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 16:58:28.927888 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.927868    2556 fs.go:135] Filesystem UUIDs: map[1467e49b-a96f-4ce1-a630-439e12db6894:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c934dfc5-adea-4972-892d-7eadc928f460:/dev/nvme0n1p3]
Jul 01 16:58:28.927957 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.927887    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 16:58:28.933406 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.933290    2556 manager.go:216] Machine: {Timestamp:2026-07-01 16:58:28.931897479 +0000 UTC m=+0.305261088 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099004 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d329a17b2f77080b10f48fa8d9f21 SystemUUID:ec2d329a-17b2-f770-80b1-0f48fa8d9f21 BootID:c654aceb-fff3-4e12-9452-f587f29024be Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:53:f7:14:62:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:53:f7:14:62:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:03:04:ae:4e:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 16:58:28.933406 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.933399    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 16:58:28.933565 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.933506    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 16:58:28.936573 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.936546    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 16:58:28.936702 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.936577    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 16:58:28.937328 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.937319    2556 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 16:58:28.937367 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.937331    2556 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 16:58:28.937367 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.937340    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 16:58:28.937996 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.937986    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 16:58:28.939139 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.939126    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 16:58:28.942094 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.942082    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 16:58:28.942361 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.942346    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 16:58:28.942461 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.942450    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 16:58:28.942516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.942505    2556 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 16:58:28.942552 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.942520    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 16:58:28.942719 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.942705    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 16:58:28.943602 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.942922    2556 kubelet.go:398] "Adding apiserver pod source"
Jul 01 16:58:28.943602 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.943118    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 16:58:28.944265 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.944250    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 16:58:28.944339 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.944276    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 16:58:28.946817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.946799    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 16:58:28.948422 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.948409    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 16:58:28.948474 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.948439    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 16:58:28.949157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949146    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 16:58:28.949199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949161    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 16:58:28.949199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949168    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 16:58:28.949199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949173    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 16:58:28.949199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949182    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 16:58:28.949199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949188    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 16:58:28.949199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949196    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 16:58:28.949199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949201    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 16:58:28.949370 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949208    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 16:58:28.949370 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949213    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 16:58:28.949370 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949222    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 16:58:28.949370 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949227    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 16:58:28.949955 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949945    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 16:58:28.949955 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.949955    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 16:58:28.953061 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.953033    2556 server.go:1268] "Started kubelet"
Jul 01 16:58:28.953183 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.953138    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 16:58:28.953239 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.953204    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 16:58:28.953295 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.953271    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 16:58:28.953976 ip-10-0-141-111 systemd[1]: Started Kubernetes Kubelet.
Jul 01 16:58:28.954240 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.954220    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 16:58:28.954324 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.954298    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 16:58:28.954776 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.954615    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 16:58:28.954834 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.954772    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 16:58:28.955847 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.955833    2556 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 16:58:28.958762 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.958726    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 16:58:28.963236 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.963219    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 16:58:28.963310 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.963217    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 16:58:28.963409 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.963303    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:28.963996 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.963872    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 16:58:28.964132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964092    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 16:58:28.964132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964112    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 16:58:28.964274 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964195    2556 factory.go:55] Registering systemd factory
Jul 01 16:58:28.964274 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964210    2556 factory.go:223] Registration of the systemd container factory successfully
Jul 01 16:58:28.964274 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964230    2556 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 16:58:28.964274 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964239    2556 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 16:58:28.964773 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964753    2556 factory.go:153] Registering CRI-O factory
Jul 01 16:58:28.964848 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964776    2556 factory.go:223] Registration of the crio container factory successfully
Jul 01 16:58:28.964848 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.963826    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-111.ec2.internal.18be38f0b444c85d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-111.ec2.internal,UID:ip-10-0-141-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-111.ec2.internal,},FirstTimestamp:2026-07-01 16:58:28.953008221 +0000 UTC m=+0.326371831,LastTimestamp:2026-07-01 16:58:28.953008221 +0000 UTC m=+0.326371831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-111.ec2.internal,}"
Jul 01 16:58:28.964848 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964822    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 16:58:28.964848 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964849    2556 factory.go:103] Registering Raw factory
Jul 01 16:58:28.965052 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.964864    2556 manager.go:1196] Started watching for new ooms in manager
Jul 01 16:58:28.965298 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.965285    2556 manager.go:349] Starting recovery of all containers
Jul 01 16:58:28.975927 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.975885    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 16:58:28.976057 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.975950    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 16:58:28.978455 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.978440    2556 manager.go:354] Recovery completed
Jul 01 16:58:28.982310 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.982294    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sx2hw"
Jul 01 16:58:28.982835 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.982822    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:28.984822 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.984805    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:28.984908 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.984838    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:28.984908 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.984852    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:28.985432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.985417    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 16:58:28.985432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.985429    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 16:58:28.985432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.985431    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:28.985592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.985444    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 16:58:28.986184 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.986111    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:28.986184 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.986153    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:28.986184 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.986168    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:28.986636 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.986622    2556 policy_none.go:49] "None policy: Start"
Jul 01 16:58:28.986691 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.986638    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 16:58:28.986691 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.986649    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 16:58:28.987733 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.987722    2556 policy_none.go:47] "Start"
Jul 01 16:58:28.993443 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:28.993426    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sx2hw"
Jul 01 16:58:28.995630 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.995551    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-111.ec2.internal.18be38f0b62a37d3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-111.ec2.internal,UID:ip-10-0-141-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-111.ec2.internal,},FirstTimestamp:2026-07-01 16:58:28.984821715 +0000 UTC m=+0.358185333,LastTimestamp:2026-07-01 16:58:28.984821715 +0000 UTC m=+0.358185333,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-111.ec2.internal,}"
Jul 01 16:58:28.998182 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:28.998119    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-111.ec2.internal.18be38f0b62a90b4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-111.ec2.internal,UID:ip-10-0-141-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-111.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-111.ec2.internal,},FirstTimestamp:2026-07-01 16:58:28.984844468 +0000 UTC m=+0.358208079,LastTimestamp:2026-07-01 16:58:28.984844468 +0000 UTC m=+0.358208079,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-111.ec2.internal,}"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.054816    2556 manager.go:337] "Starting Device Plugin manager"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.054852    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.054865    2556 server.go:87] "Starting device plugin registration server"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.055142    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.055152    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.055265    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.056104    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.056117    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.056123    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.056174    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.066277    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.067489    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.067532    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.067559    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 16:58:30.069516 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.067605    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 16:58:30.070354 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.070331    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:30.156237 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.156192    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:30.157372 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.157354    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:30.157439 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.157401    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:30.157439 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.157417    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:30.157516 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.157453    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.168687 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.168634    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal"]
Jul 01 16:58:30.168754 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.168705    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:30.170191 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.170172    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:30.170245 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.170203    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:30.170245 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.170212    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:30.171443 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.171431    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:30.171582 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.171567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.171639 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.171596    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:30.172148 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.172122    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:30.172148 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.172135    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:30.172148 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.172151    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:30.172298 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.172157    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:30.172298 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.172162    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:30.172298 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.172167    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:30.173491 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.173477    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.173554 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.173509    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:30.173743 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.173728    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.173807 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.173749    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-111.ec2.internal\": node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.174214 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.174199    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:30.174288 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.174221    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:30.174288 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.174235    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:30.193615 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.193592    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-111.ec2.internal\" not found" node="ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.196153 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.196137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e23f4cfb485cd3f723dfe6747c447460-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal\" (UID: \"e23f4cfb485cd3f723dfe6747c447460\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.196228 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.196160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e23f4cfb485cd3f723dfe6747c447460-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal\" (UID: \"e23f4cfb485cd3f723dfe6747c447460\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.196228 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.196178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80ecdb78f14d99528a812fe67731e2c2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-111.ec2.internal\" (UID: \"80ecdb78f14d99528a812fe67731e2c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.198102 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.198084    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-111.ec2.internal\" not found" node="ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.209812 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.209797    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.296611 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.296583    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e23f4cfb485cd3f723dfe6747c447460-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal\" (UID: \"e23f4cfb485cd3f723dfe6747c447460\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.296611 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.296614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e23f4cfb485cd3f723dfe6747c447460-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal\" (UID: \"e23f4cfb485cd3f723dfe6747c447460\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.296790 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.296631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80ecdb78f14d99528a812fe67731e2c2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-111.ec2.internal\" (UID: \"80ecdb78f14d99528a812fe67731e2c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.296790 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.296688    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e23f4cfb485cd3f723dfe6747c447460-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal\" (UID: \"e23f4cfb485cd3f723dfe6747c447460\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.296790 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.296714    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e23f4cfb485cd3f723dfe6747c447460-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal\" (UID: \"e23f4cfb485cd3f723dfe6747c447460\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.296790 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.296693    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/80ecdb78f14d99528a812fe67731e2c2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-111.ec2.internal\" (UID: \"80ecdb78f14d99528a812fe67731e2c2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.310439 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.310418    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.410765 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.410729    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.497419 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.497358    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.502582 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.502564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:30.511180 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.511163    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.611662 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.611624    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.712223 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.712191    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.812884 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.812803    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.913188 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:30.913159    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:30.917757 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.917736    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:30.933275 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.933259    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:30.940340 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.940325    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 16:58:30.940463 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.940440    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 16:58:30.940520 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.940462    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 16:58:30.940520 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:30.940480    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 16:58:31.013748 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:31.013715    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:31.028886 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.028853    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 16:53:30 +0000 UTC" deadline="2028-03-31 14:44:55.499985548 +0000 UTC"
Jul 01 16:58:31.028886 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.028882    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15333h46m24.471107017s"
Jul 01 16:58:31.114197 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:31.114129    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:31.135985 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:31.135947    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ecdb78f14d99528a812fe67731e2c2.slice/crio-06d2af66e0d373a4d8122b32540829c9f961450312ab44e8c2321a7d455a1256 WatchSource:0}: Error finding container 06d2af66e0d373a4d8122b32540829c9f961450312ab44e8c2321a7d455a1256: Status 404 returned error can't find the container with id 06d2af66e0d373a4d8122b32540829c9f961450312ab44e8c2321a7d455a1256
Jul 01 16:58:31.141269 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.141256    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 16:58:31.154408 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.154389    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 16:58:31.155318 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.155298    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 16:58:31.155405 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.155330    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 16:58:31.155405 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.155340    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeHasSufficientPID"
Jul 01 16:58:31.158745 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:31.158724    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23f4cfb485cd3f723dfe6747c447460.slice/crio-51e590ab069c256b9636883bd71b5c54b85ca1f237177364d65ffb43b7b8a5c4 WatchSource:0}: Error finding container 51e590ab069c256b9636883bd71b5c54b85ca1f237177364d65ffb43b7b8a5c4: Status 404 returned error can't find the container with id 51e590ab069c256b9636883bd71b5c54b85ca1f237177364d65ffb43b7b8a5c4
Jul 01 16:58:31.214369 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:31.214335    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:31.314851 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:31.314808    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:31.415350 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:31.415285    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:31.516138 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:31.516107    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-111.ec2.internal\" not found"
Jul 01 16:58:31.563264 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.563237    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:31.594608 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.594580    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:31.605939 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.605903    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 16:58:31.606917 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.606894    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal"
Jul 01 16:58:31.623855 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.623832    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 16:58:31.976480 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.976443    2556 apiserver.go:52] "Watching apiserver"
Jul 01 16:58:31.985293 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.985267    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 16:58:31.985727 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.985701    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-ddgms","kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mxpk2","openshift-dns/node-resolver-hq7vh","openshift-image-registry/node-ca-f6pw8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal","openshift-multus/multus-additional-cni-plugins-fwnqp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h","openshift-multus/multus-c8c6g","openshift-multus/network-metrics-daemon-chkmv","openshift-network-diagnostics/network-check-target-rkmmg","openshift-network-operator/iptables-alerter-6phpp","openshift-ovn-kubernetes/ovnkube-node-cht4l","kube-system/global-pull-secret-syncer-npk7n"]
Jul 01 16:58:31.987414 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.987395    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:31.991165 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.991142    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:31.992830 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.992806    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:31.994250 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.994233    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:31.994540 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.994501    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 16:58:31.995350 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.995328    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 16:58:31.995436 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.995364    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 16:58:31.995766 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.995748    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:31.997068 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.997044    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:31.997179 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:31.997146    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:31.998631 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.998598    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 16:58:31.998720 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.998703    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 16:58:31.998783 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.998740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:31.998986 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.998968    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 16:58:31.999146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999126    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vpd46\""
Jul 01 16:58:31.999146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999139    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-68dxp\""
Jul 01 16:58:31.999241 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999175    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 16:58:31.999241 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999211    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 16:58:31.999378 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999357    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 16:58:31.999378 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999367    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 16:58:31.999465 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999447    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 16:58:31.999522 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999368    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x2l9q\""
Jul 01 16:58:31.999522 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999486    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 16:58:31.999618 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999600    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 16:58:31.999657 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999618    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 16:58:31.999657 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999651    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 16:58:31.999738 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:31.999726    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 16:58:32.000167 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.000144    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ttdvv\""
Jul 01 16:58:32.000929 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.000910    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.002544 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.002516    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.003990 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.003891    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rqcbm\""
Jul 01 16:58:32.003990 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.003985    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p8twl\""
Jul 01 16:58:32.004166 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.004091    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:32.004235 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.004162    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:32.004382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.004365    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 16:58:32.004439 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.004412    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 16:58:32.004559 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.004545    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-systemd\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-env-overrides\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005370    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-var-lib-kubelet\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005395    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-os-release\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005465    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005483    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg85k\" (UniqueName: \"kubernetes.io/projected/067f479a-c3e1-4a2e-badc-f863f901a14b-kube-api-access-tg85k\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005511    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-systemd\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-run\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005588    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-var-lib-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005631    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-node-log\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005652    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-cni-bin\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005684    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-cni-netd\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.005817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005710    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f0b617c-0bf7-4462-a62b-2aee12c53d78-serviceca\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005726    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005735    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-system-cni-dir\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005793    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005876    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005879    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-kubelet\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005952    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-run-netns\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.005988    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/067f479a-c3e1-4a2e-badc-f863f901a14b-ovn-node-metrics-cert\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006028    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-lib-modules\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006074    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-modprobe-d\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006082    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xssw6\""
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006109    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysctl-d\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006112    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-ovn\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.006174    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006177    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysctl-conf\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006214    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-tmp-dir\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006180    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x6s5b\""
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006248    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 16:58:32.006592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006260    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhxx\" (UniqueName: \"kubernetes.io/projected/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-kube-api-access-plhxx\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhh4\" (UniqueName: \"kubernetes.io/projected/1f0b617c-0bf7-4462-a62b-2aee12c53d78-kube-api-access-ckhh4\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006344    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-systemd-units\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-log-socket\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006811    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysconfig\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-kubernetes\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006882    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb6609e-f517-4387-be54-bececf7e7958-tmp\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006913    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.006940    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-ovnkube-config\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-host\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007100    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sbc\" (UniqueName: \"kubernetes.io/projected/7fb6609e-f517-4387-be54-bececf7e7958-kube-api-access-98sbc\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-cnibin\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dntl\" (UniqueName: \"kubernetes.io/projected/c53c6577-4cf4-45a2-802a-690ea2560a57-kube-api-access-2dntl\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.007434 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-etc-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.008089 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007709    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.008089 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007714    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-ovnkube-script-lib\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.008089 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-sys\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.008089 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.007946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fb6609e-f517-4387-be54-bececf7e7958-etc-tuned\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.008089 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.008026    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-hosts-file\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.008319 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.008257    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f0b617c-0bf7-4462-a62b-2aee12c53d78-host\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.008319 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.008303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.008415 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.008338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-slash\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.010833 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.010812    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 16:58:32.010920 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.010908    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wz8r8\""
Jul 01 16:58:32.011046 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.011030    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 16:58:32.011708 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.011678    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 16:58:32.020622 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.020597    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 16:58:32.063243 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.063216    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rs9fn"
Jul 01 16:58:32.072650 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.072577    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal" event={"ID":"e23f4cfb485cd3f723dfe6747c447460","Type":"ContainerStarted","Data":"51e590ab069c256b9636883bd71b5c54b85ca1f237177364d65ffb43b7b8a5c4"}
Jul 01 16:58:32.073771 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.073745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal" event={"ID":"80ecdb78f14d99528a812fe67731e2c2","Type":"ContainerStarted","Data":"06d2af66e0d373a4d8122b32540829c9f961450312ab44e8c2321a7d455a1256"}
Jul 01 16:58:32.083268 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.083250    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rs9fn"
Jul 01 16:58:32.095939 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.095918    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 16:58:32.108853 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.108827    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-hosts-file\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.108969 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.108866    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f0b617c-0bf7-4462-a62b-2aee12c53d78-host\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.108969 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.108894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-slash\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.108969 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.108918    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-env-overrides\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.109146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.108971    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f0b617c-0bf7-4462-a62b-2aee12c53d78-host\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.109146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109007    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-var-lib-kubelet\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.109146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg85k\" (UniqueName: \"kubernetes.io/projected/067f479a-c3e1-4a2e-badc-f863f901a14b-kube-api-access-tg85k\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.109146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109078    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvf7t\" (UniqueName: \"kubernetes.io/projected/d271da50-b129-4761-9ece-af5b472a97e8-kube-api-access-jvf7t\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.109146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-device-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.109146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-sys-fs\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-systemd\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109178    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-run\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109202    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/067f479a-c3e1-4a2e-badc-f863f901a14b-ovn-node-metrics-cert\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-netns\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109274    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-socket-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109298    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-system-cni-dir\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-kubelet\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109346    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-registration-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-modprobe-d\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109397    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysctl-d\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109420    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-etc-kubernetes\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.109432 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109426    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-env-overrides\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmdp\" (UniqueName: \"kubernetes.io/projected/406c4f26-829c-4bbc-b670-2adde865c36f-kube-api-access-jvmdp\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysctl-conf\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109565    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-slash\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109606    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-hosts-file\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109623    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-modprobe-d\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109644    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-var-lib-kubelet\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109686    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-systemd-units\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-log-socket\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109717    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysctl-d\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-os-release\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109756    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-system-cni-dir\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109758    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-run\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb6609e-f517-4387-be54-bececf7e7958-tmp\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109817    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-systemd\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109894    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysctl-conf\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109923    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-kubelet\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.109954    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-systemd-units\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.110049 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110001    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-log-socket\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110055    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-socket-dir-parent\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110168    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sbc\" (UniqueName: \"kubernetes.io/projected/7fb6609e-f517-4387-be54-bececf7e7958-kube-api-access-98sbc\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110196    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-cnibin\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110238    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dntl\" (UniqueName: \"kubernetes.io/projected/c53c6577-4cf4-45a2-802a-690ea2560a57-kube-api-access-2dntl\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110406    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40127afb-5366-4e79-9f41-64186c92e106-dbus\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110441    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110463    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-cnibin\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-cni-bin\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110500    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-cnibin\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110512    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d271da50-b129-4761-9ece-af5b472a97e8-multus-daemon-config\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-sys\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110585    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-sys\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fb6609e-f517-4387-be54-bececf7e7958-etc-tuned\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.110889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-systemd\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-multus-certs\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110758    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-systemd\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-os-release\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110830    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110886    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-run-ovn-kubernetes\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110901    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-system-cni-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d271da50-b129-4761-9ece-af5b472a97e8-cni-binary-copy\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110932    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110937    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c53c6577-4cf4-45a2-802a-690ea2560a57-os-release\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110952    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f18c7204-9546-4e24-ae93-4edb51af02dd-iptables-alerter-script\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.111758 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.110992    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-var-lib-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111079    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-node-log\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111111    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-var-lib-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111135    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-cni-bin\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111160    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-cni-netd\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111173    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f0b617c-0bf7-4462-a62b-2aee12c53d78-serviceca\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111210    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111224    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-cni-bin\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-run-netns\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111259    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40127afb-5366-4e79-9f41-64186c92e106-kubelet-config\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111271    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-node-log\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8e605ac-da6b-40ed-8fa6-079d387ea570-agent-certs\") pod \"konnectivity-agent-ddgms\" (UID: \"b8e605ac-da6b-40ed-8fa6-079d387ea570\") " pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-lib-modules\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111326    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.112518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-ovn\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-cni-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111378    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-cni-netd\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111394    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tjj\" (UniqueName: \"kubernetes.io/projected/8e950e64-9263-493a-8a07-39043d185975-kube-api-access-64tjj\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111417    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f18c7204-9546-4e24-ae93-4edb51af02dd-host-slash\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111442    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-tmp-dir\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111435    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-host-run-netns\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhxx\" (UniqueName: \"kubernetes.io/projected/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-kube-api-access-plhxx\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhh4\" (UniqueName: \"kubernetes.io/projected/1f0b617c-0bf7-4462-a62b-2aee12c53d78-kube-api-access-ckhh4\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111517    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-run-ovn\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111526    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-cni-multus\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111570    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-kubelet\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111610    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-conf-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111637    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8e605ac-da6b-40ed-8fa6-079d387ea570-konnectivity-ca\") pod \"konnectivity-agent-ddgms\" (UID: \"b8e605ac-da6b-40ed-8fa6-079d387ea570\") " pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111689    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysconfig\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111692    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-kubernetes\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.113325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-ovnkube-config\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111771    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c53c6577-4cf4-45a2-802a-690ea2560a57-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.111933    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1f0b617c-0bf7-4462-a62b-2aee12c53d78-serviceca\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112048    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-lib-modules\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112048    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-tmp-dir\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112091    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-k8s-cni-cncf-io\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112120    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-host\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112208    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-host\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112244    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-sysconfig\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112262    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-ovnkube-config\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-etc-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112313    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7fb6609e-f517-4387-be54-bececf7e7958-etc-kubernetes\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-ovnkube-script-lib\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112360    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/067f479a-c3e1-4a2e-badc-f863f901a14b-etc-openvswitch\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-hostroot\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112486    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96c6p\" (UniqueName: \"kubernetes.io/projected/f18c7204-9546-4e24-ae93-4edb51af02dd-kube-api-access-96c6p\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.114132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.112844    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/067f479a-c3e1-4a2e-badc-f863f901a14b-ovnkube-script-lib\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.114855 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.113738    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/067f479a-c3e1-4a2e-badc-f863f901a14b-ovn-node-metrics-cert\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.114855 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.114154    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7fb6609e-f517-4387-be54-bececf7e7958-etc-tuned\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.114855 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.114185    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb6609e-f517-4387-be54-bececf7e7958-tmp\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.128838 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.128816    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg85k\" (UniqueName: \"kubernetes.io/projected/067f479a-c3e1-4a2e-badc-f863f901a14b-kube-api-access-tg85k\") pod \"ovnkube-node-cht4l\" (UID: \"067f479a-c3e1-4a2e-badc-f863f901a14b\") " pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.131696 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.131649    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dntl\" (UniqueName: \"kubernetes.io/projected/c53c6577-4cf4-45a2-802a-690ea2560a57-kube-api-access-2dntl\") pod \"multus-additional-cni-plugins-fwnqp\" (UID: \"c53c6577-4cf4-45a2-802a-690ea2560a57\") " pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.133201 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.133175    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhh4\" (UniqueName: \"kubernetes.io/projected/1f0b617c-0bf7-4462-a62b-2aee12c53d78-kube-api-access-ckhh4\") pod \"node-ca-f6pw8\" (UID: \"1f0b617c-0bf7-4462-a62b-2aee12c53d78\") " pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.134033 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.134010    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sbc\" (UniqueName: \"kubernetes.io/projected/7fb6609e-f517-4387-be54-bececf7e7958-kube-api-access-98sbc\") pod \"tuned-mxpk2\" (UID: \"7fb6609e-f517-4387-be54-bececf7e7958\") " pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.134765 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.134679    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhxx\" (UniqueName: \"kubernetes.io/projected/b277c8b7-1860-4ca6-97db-9e64e6f2ec9d-kube-api-access-plhxx\") pod \"node-resolver-hq7vh\" (UID: \"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d\") " pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.212971 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.212930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-registration-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.212983    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-etc-kubernetes\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213013    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmdp\" (UniqueName: \"kubernetes.io/projected/406c4f26-829c-4bbc-b670-2adde865c36f-kube-api-access-jvmdp\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-os-release\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213080    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-socket-dir-parent\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213106    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40127afb-5366-4e79-9f41-64186c92e106-dbus\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213128    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-etc-kubernetes\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213157 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-cnibin\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213174    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-cni-bin\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213184    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-registration-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213205    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d271da50-b129-4761-9ece-af5b472a97e8-multus-daemon-config\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213215    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-socket-dir-parent\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213236    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-multus-certs\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213257    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.213310    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-system-cni-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d271da50-b129-4761-9ece-af5b472a97e8-cni-binary-copy\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213366    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-cnibin\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f18c7204-9546-4e24-ae93-4edb51af02dd-iptables-alerter-script\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.213394    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret podName:40127afb-5366-4e79-9f41-64186c92e106 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:32.713371982 +0000 UTC m=+3.055712622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret") pod "global-pull-secret-syncer-npk7n" (UID: "40127afb-5366-4e79-9f41-64186c92e106") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213426    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/40127afb-5366-4e79-9f41-64186c92e106-dbus\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213442    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40127afb-5366-4e79-9f41-64186c92e106-kubelet-config\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.213546 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8e605ac-da6b-40ed-8fa6-079d387ea570-agent-certs\") pod \"konnectivity-agent-ddgms\" (UID: \"b8e605ac-da6b-40ed-8fa6-079d387ea570\") " pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213478    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-cni-bin\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213499    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-cni-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tjj\" (UniqueName: \"kubernetes.io/projected/8e950e64-9263-493a-8a07-39043d185975-kube-api-access-64tjj\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213539    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-os-release\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213542    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213562    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f18c7204-9546-4e24-ae93-4edb51af02dd-host-slash\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213576    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-multus-certs\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-cni-multus\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213623    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/40127afb-5366-4e79-9f41-64186c92e106-kubelet-config\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213624    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-cni-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-kubelet\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.213654    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213665    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-kubelet\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213671    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-system-cni-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213701    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-var-lib-cni-multus\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.213717    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs podName:406c4f26-829c-4bbc-b670-2adde865c36f nodeName:}" failed. No retries permitted until 2026-07-01 16:58:32.713700754 +0000 UTC m=+3.056041399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs") pod "network-metrics-daemon-chkmv" (UID: "406c4f26-829c-4bbc-b670-2adde865c36f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:32.214382 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213734    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f18c7204-9546-4e24-ae93-4edb51af02dd-host-slash\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-conf-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213756    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-etc-selinux\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213770    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8e605ac-da6b-40ed-8fa6-079d387ea570-konnectivity-ca\") pod \"konnectivity-agent-ddgms\" (UID: \"b8e605ac-da6b-40ed-8fa6-079d387ea570\") " pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213792    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-multus-conf-dir\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213798    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-k8s-cni-cncf-io\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213829    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-hostroot\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213856    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96c6p\" (UniqueName: \"kubernetes.io/projected/f18c7204-9546-4e24-ae93-4edb51af02dd-kube-api-access-96c6p\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213909    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-hostroot\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213918    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvf7t\" (UniqueName: \"kubernetes.io/projected/d271da50-b129-4761-9ece-af5b472a97e8-kube-api-access-jvf7t\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213942    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-device-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213947    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f18c7204-9546-4e24-ae93-4edb51af02dd-iptables-alerter-script\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213947    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d271da50-b129-4761-9ece-af5b472a97e8-multus-daemon-config\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213951    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-k8s-cni-cncf-io\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.213977    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-sys-fs\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-netns\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215140 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214024    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-socket-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.215792 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214114    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d271da50-b129-4761-9ece-af5b472a97e8-cni-binary-copy\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215792 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214142    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-device-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.215792 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214160    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-sys-fs\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.215792 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214163    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e950e64-9263-493a-8a07-39043d185975-socket-dir\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.215792 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214164    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d271da50-b129-4761-9ece-af5b472a97e8-host-run-netns\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.215792 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.214333    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8e605ac-da6b-40ed-8fa6-079d387ea570-konnectivity-ca\") pod \"konnectivity-agent-ddgms\" (UID: \"b8e605ac-da6b-40ed-8fa6-079d387ea570\") " pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.216383 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.216361    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8e605ac-da6b-40ed-8fa6-079d387ea570-agent-certs\") pod \"konnectivity-agent-ddgms\" (UID: \"b8e605ac-da6b-40ed-8fa6-079d387ea570\") " pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.231648 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.231573    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:32.231648 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.231600    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:32.231648 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.231616    2556 projected.go:196] Error preparing data for projected volume kube-api-access-bw22c for pod openshift-network-diagnostics/network-check-target-rkmmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:32.231894 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.231693    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmdp\" (UniqueName: \"kubernetes.io/projected/406c4f26-829c-4bbc-b670-2adde865c36f-kube-api-access-jvmdp\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:32.231894 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.231707    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c podName:0f425d31-4437-4e8d-bd20-3c3e40bdb24b nodeName:}" failed. No retries permitted until 2026-07-01 16:58:32.731689401 +0000 UTC m=+3.074030039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bw22c" (UniqueName: "kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c") pod "network-check-target-rkmmg" (UID: "0f425d31-4437-4e8d-bd20-3c3e40bdb24b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:32.233537 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.233475    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96c6p\" (UniqueName: \"kubernetes.io/projected/f18c7204-9546-4e24-ae93-4edb51af02dd-kube-api-access-96c6p\") pod \"iptables-alerter-6phpp\" (UID: \"f18c7204-9546-4e24-ae93-4edb51af02dd\") " pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.234413 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.234394    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvf7t\" (UniqueName: \"kubernetes.io/projected/d271da50-b129-4761-9ece-af5b472a97e8-kube-api-access-jvf7t\") pod \"multus-c8c6g\" (UID: \"d271da50-b129-4761-9ece-af5b472a97e8\") " pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.234642 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.234620    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tjj\" (UniqueName: \"kubernetes.io/projected/8e950e64-9263-493a-8a07-39043d185975-kube-api-access-64tjj\") pod \"aws-ebs-csi-driver-node-gmv4h\" (UID: \"8e950e64-9263-493a-8a07-39043d185975\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.303011 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.302980    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwnqp"
Jul 01 16:58:32.310600 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.310416    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:32.319598 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.319571    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mxpk2"
Jul 01 16:58:32.324403 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.324383    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hq7vh"
Jul 01 16:58:32.338530 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.338509    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f6pw8"
Jul 01 16:58:32.346373 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.346354    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c8c6g"
Jul 01 16:58:32.353343 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.353288    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:32.362290 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.362266    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h"
Jul 01 16:58:32.369557 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.369513    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6phpp"
Jul 01 16:58:32.515990 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.515900    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:32.717731 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.717698    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:32.717919 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.717739    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:32.717919 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.717868    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:32.718023 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.717927    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs podName:406c4f26-829c-4bbc-b670-2adde865c36f nodeName:}" failed. No retries permitted until 2026-07-01 16:58:33.71791379 +0000 UTC m=+4.060254432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs") pod "network-metrics-daemon-chkmv" (UID: "406c4f26-829c-4bbc-b670-2adde865c36f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:32.718023 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.717869    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:32.718139 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.718032    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret podName:40127afb-5366-4e79-9f41-64186c92e106 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:33.718013331 +0000 UTC m=+4.060353969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret") pod "global-pull-secret-syncer-npk7n" (UID: "40127afb-5366-4e79-9f41-64186c92e106") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:32.818843 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:32.818756    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:32.818995 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.818944    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:32.818995 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.818970    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:32.818995 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.818982    2556 projected.go:196] Error preparing data for projected volume kube-api-access-bw22c for pod openshift-network-diagnostics/network-check-target-rkmmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:32.819160 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:32.819039    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c podName:0f425d31-4437-4e8d-bd20-3c3e40bdb24b nodeName:}" failed. No retries permitted until 2026-07-01 16:58:33.819024226 +0000 UTC m=+4.161364883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bw22c" (UniqueName: "kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c") pod "network-check-target-rkmmg" (UID: "0f425d31-4437-4e8d-bd20-3c3e40bdb24b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:32.987794 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:32.987699    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb6609e_f517_4387_be54_bececf7e7958.slice/crio-766b4e9932cfc73fc4f5316c551f00cf09c94a2b3f491bbe0b183e2b567d4f74 WatchSource:0}: Error finding container 766b4e9932cfc73fc4f5316c551f00cf09c94a2b3f491bbe0b183e2b567d4f74: Status 404 returned error can't find the container with id 766b4e9932cfc73fc4f5316c551f00cf09c94a2b3f491bbe0b183e2b567d4f74
Jul 01 16:58:32.990695 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:32.990528    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0b617c_0bf7_4462_a62b_2aee12c53d78.slice/crio-62d87c264dfbc1a25e4bb3ba6735f6c608dd095e1415540ac3e556c2fdfdec9a WatchSource:0}: Error finding container 62d87c264dfbc1a25e4bb3ba6735f6c608dd095e1415540ac3e556c2fdfdec9a: Status 404 returned error can't find the container with id 62d87c264dfbc1a25e4bb3ba6735f6c608dd095e1415540ac3e556c2fdfdec9a
Jul 01 16:58:32.996307 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:32.996282    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e605ac_da6b_40ed_8fa6_079d387ea570.slice/crio-fd1096d3f44b60282e5699b37307cbfb26df3c3dd3cb1be4fe0aefbbf3b81b9b WatchSource:0}: Error finding container fd1096d3f44b60282e5699b37307cbfb26df3c3dd3cb1be4fe0aefbbf3b81b9b: Status 404 returned error can't find the container with id fd1096d3f44b60282e5699b37307cbfb26df3c3dd3cb1be4fe0aefbbf3b81b9b
Jul 01 16:58:32.997198 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:32.997179    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067f479a_c3e1_4a2e_badc_f863f901a14b.slice/crio-7bb575a83114e18dca24e61bc470ef0014513508b7aa3735eb44d8f545fb3155 WatchSource:0}: Error finding container 7bb575a83114e18dca24e61bc470ef0014513508b7aa3735eb44d8f545fb3155: Status 404 returned error can't find the container with id 7bb575a83114e18dca24e61bc470ef0014513508b7aa3735eb44d8f545fb3155
Jul 01 16:58:32.997744 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:32.997726    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18c7204_9546_4e24_ae93_4edb51af02dd.slice/crio-04b8542fd6394d63220fa7f410b027fb535d5df8bd19f192207c4c1e72a4d517 WatchSource:0}: Error finding container 04b8542fd6394d63220fa7f410b027fb535d5df8bd19f192207c4c1e72a4d517: Status 404 returned error can't find the container with id 04b8542fd6394d63220fa7f410b027fb535d5df8bd19f192207c4c1e72a4d517
Jul 01 16:58:32.999830 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:32.999802    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e950e64_9263_493a_8a07_39043d185975.slice/crio-1becbbe9d13cf488732cf427547b76b24234327702f6007afcca1a71d8b64dd8 WatchSource:0}: Error finding container 1becbbe9d13cf488732cf427547b76b24234327702f6007afcca1a71d8b64dd8: Status 404 returned error can't find the container with id 1becbbe9d13cf488732cf427547b76b24234327702f6007afcca1a71d8b64dd8
Jul 01 16:58:33.001055 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:33.000539    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53c6577_4cf4_45a2_802a_690ea2560a57.slice/crio-9663ff3931a9ca131ba6f91c88786d537e6c6ce995f08d877565f33bf208b3d3 WatchSource:0}: Error finding container 9663ff3931a9ca131ba6f91c88786d537e6c6ce995f08d877565f33bf208b3d3: Status 404 returned error can't find the container with id 9663ff3931a9ca131ba6f91c88786d537e6c6ce995f08d877565f33bf208b3d3
Jul 01 16:58:33.001399 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:33.001193    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb277c8b7_1860_4ca6_97db_9e64e6f2ec9d.slice/crio-8bee27f30ab7d2584bf31667e4ad96d7c70cae49870a394b1d02b80a7ca151b3 WatchSource:0}: Error finding container 8bee27f30ab7d2584bf31667e4ad96d7c70cae49870a394b1d02b80a7ca151b3: Status 404 returned error can't find the container with id 8bee27f30ab7d2584bf31667e4ad96d7c70cae49870a394b1d02b80a7ca151b3
Jul 01 16:58:33.002338 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:58:33.002112    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd271da50_b129_4761_9ece_af5b472a97e8.slice/crio-d1cb69ef20d789bd6e11ba6b8dffec35a131e7c8843215282670c5b523ef99f9 WatchSource:0}: Error finding container d1cb69ef20d789bd6e11ba6b8dffec35a131e7c8843215282670c5b523ef99f9: Status 404 returned error can't find the container with id d1cb69ef20d789bd6e11ba6b8dffec35a131e7c8843215282670c5b523ef99f9
Jul 01 16:58:33.006527 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.004623    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:33.067835 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.067816    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:33.067950 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.067930    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:33.077475 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.077401    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c8c6g" event={"ID":"d271da50-b129-4761-9ece-af5b472a97e8","Type":"ContainerStarted","Data":"d1cb69ef20d789bd6e11ba6b8dffec35a131e7c8843215282670c5b523ef99f9"}
Jul 01 16:58:33.078388 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.078367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6phpp" event={"ID":"f18c7204-9546-4e24-ae93-4edb51af02dd","Type":"ContainerStarted","Data":"04b8542fd6394d63220fa7f410b027fb535d5df8bd19f192207c4c1e72a4d517"}
Jul 01 16:58:33.079468 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.079447    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"7bb575a83114e18dca24e61bc470ef0014513508b7aa3735eb44d8f545fb3155"}
Jul 01 16:58:33.080421 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.080401    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ddgms" event={"ID":"b8e605ac-da6b-40ed-8fa6-079d387ea570","Type":"ContainerStarted","Data":"fd1096d3f44b60282e5699b37307cbfb26df3c3dd3cb1be4fe0aefbbf3b81b9b"}
Jul 01 16:58:33.081479 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.081456    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mxpk2" event={"ID":"7fb6609e-f517-4387-be54-bececf7e7958","Type":"ContainerStarted","Data":"766b4e9932cfc73fc4f5316c551f00cf09c94a2b3f491bbe0b183e2b567d4f74"}
Jul 01 16:58:33.082401 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.082383    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hq7vh" event={"ID":"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d","Type":"ContainerStarted","Data":"8bee27f30ab7d2584bf31667e4ad96d7c70cae49870a394b1d02b80a7ca151b3"}
Jul 01 16:58:33.083466 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.083425    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerStarted","Data":"9663ff3931a9ca131ba6f91c88786d537e6c6ce995f08d877565f33bf208b3d3"}
Jul 01 16:58:33.084562 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.084536    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h" event={"ID":"8e950e64-9263-493a-8a07-39043d185975","Type":"ContainerStarted","Data":"1becbbe9d13cf488732cf427547b76b24234327702f6007afcca1a71d8b64dd8"}
Jul 01 16:58:33.085344 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.085298    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 16:53:32 +0000 UTC" deadline="2028-04-17 09:40:28.400797474 +0000 UTC"
Jul 01 16:58:33.085425 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.085345    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15736h41m55.315456853s"
Jul 01 16:58:33.085542 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.085513    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6pw8" event={"ID":"1f0b617c-0bf7-4462-a62b-2aee12c53d78","Type":"ContainerStarted","Data":"62d87c264dfbc1a25e4bb3ba6735f6c608dd095e1415540ac3e556c2fdfdec9a"}
Jul 01 16:58:33.725359 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.725320    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:33.725575 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.725376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:33.725575 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.725528    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:33.725688 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.725590    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs podName:406c4f26-829c-4bbc-b670-2adde865c36f nodeName:}" failed. No retries permitted until 2026-07-01 16:58:35.725571923 +0000 UTC m=+6.067912564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs") pod "network-metrics-daemon-chkmv" (UID: "406c4f26-829c-4bbc-b670-2adde865c36f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:33.725688 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.725661    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:33.725817 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.725694    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret podName:40127afb-5366-4e79-9f41-64186c92e106 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:35.725683577 +0000 UTC m=+6.068024222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret") pod "global-pull-secret-syncer-npk7n" (UID: "40127afb-5366-4e79-9f41-64186c92e106") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:33.826545 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.826383    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:33.826708 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.826619    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:33.826708 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.826640    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:33.826708 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.826652    2556 projected.go:196] Error preparing data for projected volume kube-api-access-bw22c for pod openshift-network-diagnostics/network-check-target-rkmmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:33.826868 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:33.826718    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c podName:0f425d31-4437-4e8d-bd20-3c3e40bdb24b nodeName:}" failed. No retries permitted until 2026-07-01 16:58:35.826699857 +0000 UTC m=+6.169040496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bw22c" (UniqueName: "kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c") pod "network-check-target-rkmmg" (UID: "0f425d31-4437-4e8d-bd20-3c3e40bdb24b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:33.846772 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:33.846736    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 16:58:34.071301 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.070565    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:34.071301 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:34.070698    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:34.071301 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.071112    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:34.071301 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:34.071223    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:34.086125 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.086032    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 16:53:32 +0000 UTC" deadline="2028-03-05 16:08:37.548628576 +0000 UTC"
Jul 01 16:58:34.086125 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.086083    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14711h10m3.462550832s"
Jul 01 16:58:34.114957 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.114867    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal" event={"ID":"80ecdb78f14d99528a812fe67731e2c2","Type":"ContainerStarted","Data":"f4d4674b4465471442d0ed962238947aa3e14cf5dcbf86542fcb86c6ee181500"}
Jul 01 16:58:34.122935 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.122902    2556 generic.go:356] "Generic (PLEG): container finished" podID="e23f4cfb485cd3f723dfe6747c447460" containerID="d942e01dad40f40f48278047586b60c834af1ea828cda55d350bf9258097d605" exitCode=0
Jul 01 16:58:34.123115 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.122950    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal" event={"ID":"e23f4cfb485cd3f723dfe6747c447460","Type":"ContainerDied","Data":"d942e01dad40f40f48278047586b60c834af1ea828cda55d350bf9258097d605"}
Jul 01 16:58:34.131853 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:34.131540    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-111.ec2.internal" podStartSLOduration=3.131510434 podStartE2EDuration="3.131510434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:34.130902719 +0000 UTC m=+4.473243379" watchObservedRunningTime="2026-07-01 16:58:34.131510434 +0000 UTC m=+4.473851098"
Jul 01 16:58:35.068201 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:35.068166    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:35.068389 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.068296    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:35.143508 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:35.143135    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal" event={"ID":"e23f4cfb485cd3f723dfe6747c447460","Type":"ContainerStarted","Data":"641d59b1667739f6b72f3d730f2ead88006ba379b476b6899fc90a9a6d003a24"}
Jul 01 16:58:35.163923 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:35.163722    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-111.ec2.internal" podStartSLOduration=4.163701337 podStartE2EDuration="4.163701337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:35.16263429 +0000 UTC m=+5.504974951" watchObservedRunningTime="2026-07-01 16:58:35.163701337 +0000 UTC m=+5.506041999"
Jul 01 16:58:35.742192 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:35.742150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:35.742394 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:35.742211    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:35.742394 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.742379    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:35.742507 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.742446    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs podName:406c4f26-829c-4bbc-b670-2adde865c36f nodeName:}" failed. No retries permitted until 2026-07-01 16:58:39.742425842 +0000 UTC m=+10.084766480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs") pod "network-metrics-daemon-chkmv" (UID: "406c4f26-829c-4bbc-b670-2adde865c36f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:35.742965 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.742867    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:35.742965 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.742921    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret podName:40127afb-5366-4e79-9f41-64186c92e106 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:39.742905087 +0000 UTC m=+10.085245729 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret") pod "global-pull-secret-syncer-npk7n" (UID: "40127afb-5366-4e79-9f41-64186c92e106") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:35.843661 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:35.842958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:35.843661 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.843224    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:35.843661 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.843248    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:35.843661 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.843261    2556 projected.go:196] Error preparing data for projected volume kube-api-access-bw22c for pod openshift-network-diagnostics/network-check-target-rkmmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:35.843661 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:35.843319    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c podName:0f425d31-4437-4e8d-bd20-3c3e40bdb24b nodeName:}" failed. No retries permitted until 2026-07-01 16:58:39.843302866 +0000 UTC m=+10.185643523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bw22c" (UniqueName: "kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c") pod "network-check-target-rkmmg" (UID: "0f425d31-4437-4e8d-bd20-3c3e40bdb24b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:36.070144 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:36.070052    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:36.070291 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:36.070209    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:36.070359 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:36.070072    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:36.070533 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:36.070440    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:37.068242 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:37.068210    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:37.068704 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:37.068344    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:38.068370 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:38.068340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:38.068782 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:38.068477    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:38.068782 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:38.068594    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:38.068782 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:38.068718    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:39.068812 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:39.068317    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:39.068812 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.068457    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:39.775620 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:39.775558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:39.775620 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:39.775617    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:39.775878 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.775736    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:39.775878 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.775752    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:39.775878 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.775815    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret podName:40127afb-5366-4e79-9f41-64186c92e106 nodeName:}" failed. No retries permitted until 2026-07-01 16:58:47.775795118 +0000 UTC m=+18.118135759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret") pod "global-pull-secret-syncer-npk7n" (UID: "40127afb-5366-4e79-9f41-64186c92e106") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:39.775878 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.775837    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs podName:406c4f26-829c-4bbc-b670-2adde865c36f nodeName:}" failed. No retries permitted until 2026-07-01 16:58:47.775826286 +0000 UTC m=+18.118166928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs") pod "network-metrics-daemon-chkmv" (UID: "406c4f26-829c-4bbc-b670-2adde865c36f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:39.876573 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:39.876534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:39.876732 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.876707    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:39.876799 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.876734    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:39.876799 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.876747    2556 projected.go:196] Error preparing data for projected volume kube-api-access-bw22c for pod openshift-network-diagnostics/network-check-target-rkmmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:39.876906 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:39.876807    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c podName:0f425d31-4437-4e8d-bd20-3c3e40bdb24b nodeName:}" failed. No retries permitted until 2026-07-01 16:58:47.876788851 +0000 UTC m=+18.219129513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bw22c" (UniqueName: "kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c") pod "network-check-target-rkmmg" (UID: "0f425d31-4437-4e8d-bd20-3c3e40bdb24b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:40.068943 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:40.068865    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:40.069380 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:40.068975    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:40.069380 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:40.069113    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:40.069380 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:40.069241    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:41.068206 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:41.068168    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:41.068374 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:41.068312    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:42.068205 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:42.068162    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:42.068205 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:42.068202    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:42.068746 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:42.068314    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:42.068746 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:42.068365    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:43.068611 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:43.068575    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:43.069056 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:43.068693    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:44.068159 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:44.068076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:44.068313 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:44.068221    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:44.068313 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:44.068279    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:44.068440 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:44.068418    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:45.068602 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:45.068561    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:45.068981 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:45.068681    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:46.068723 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:46.068692    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:46.069183 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:46.068822    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:46.069183 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:46.068868    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:46.069183 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:46.068983    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:47.068394 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:47.068362    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:47.068663 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.068479    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:47.834641 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:47.834553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:47.834641 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:47.834605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:47.835177 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.834724    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:47.835177 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.834787    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs podName:406c4f26-829c-4bbc-b670-2adde865c36f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:03.834769153 +0000 UTC m=+34.177109796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs") pod "network-metrics-daemon-chkmv" (UID: "406c4f26-829c-4bbc-b670-2adde865c36f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:58:47.835177 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.834725    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:47.835177 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.834875    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret podName:40127afb-5366-4e79-9f41-64186c92e106 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:03.834854442 +0000 UTC m=+34.177195102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret") pod "global-pull-secret-syncer-npk7n" (UID: "40127afb-5366-4e79-9f41-64186c92e106") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:58:47.935241 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:47.935187    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:47.935433 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.935369    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:58:47.935433 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.935396    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:58:47.935433 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.935408    2556 projected.go:196] Error preparing data for projected volume kube-api-access-bw22c for pod openshift-network-diagnostics/network-check-target-rkmmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:47.935581 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:47.935475    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c podName:0f425d31-4437-4e8d-bd20-3c3e40bdb24b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:03.93545522 +0000 UTC m=+34.277795860 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bw22c" (UniqueName: "kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c") pod "network-check-target-rkmmg" (UID: "0f425d31-4437-4e8d-bd20-3c3e40bdb24b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:58:48.068363 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:48.068313    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:48.068363 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:48.068313    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:48.068608 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:48.068446    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:48.068608 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:48.068592    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:49.068920 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:49.068718    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:49.069409 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:49.069014    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:50.069123 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:50.069094    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:50.069605 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:50.069132    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:50.069605 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:50.069211    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:50.069605 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:50.069386    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:51.068728 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:51.068683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:51.068918 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:51.068813    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:52.068984 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:52.068328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:52.068984 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:52.068456    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:52.068984 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:52.068837    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:52.068984 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:52.068949    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:52.179706 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:52.179668    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c8c6g" event={"ID":"d271da50-b129-4761-9ece-af5b472a97e8","Type":"ContainerStarted","Data":"b72057d7458a213c7e9418def33d17ae35be37d39c45200df82f4ebbd57ad28d"}
Jul 01 16:58:52.181255 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:52.181232    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"fb6033ce3664c19e19896c76e485ee07fd82a6b1221908e7bf74fe4d589e6898"}
Jul 01 16:58:52.182924 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:52.182748    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mxpk2" event={"ID":"7fb6609e-f517-4387-be54-bececf7e7958","Type":"ContainerStarted","Data":"b99891664dcdae75cfac8137712631bc06880833d92d8f6ac06b669b8e908eac"}
Jul 01 16:58:52.203499 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:52.202664    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c8c6g" podStartSLOduration=3.173056404 podStartE2EDuration="22.202641468s" totalImagesPullingTime="19.029585064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:52.202438671 +0000 UTC m=+22.544779332" watchObservedRunningTime="2026-07-01 16:58:52.202641468 +0000 UTC m=+22.544982128"
Jul 01 16:58:52.228106 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:52.227776    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mxpk2" podStartSLOduration=3.245149349 podStartE2EDuration="22.227756229s" totalImagesPullingTime="18.98260688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:52.227744878 +0000 UTC m=+22.570085539" watchObservedRunningTime="2026-07-01 16:58:52.227756229 +0000 UTC m=+22.570096891"
Jul 01 16:58:53.068428 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.068188    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:53.068576 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:53.068519    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:53.185962 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.185925    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h" event={"ID":"8e950e64-9263-493a-8a07-39043d185975","Type":"ContainerStarted","Data":"4ce84e1877b586ec98a2142f153229be0906f010b37f1ef967cc78bfe8af2211"}
Jul 01 16:58:53.187188 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.187166    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f6pw8" event={"ID":"1f0b617c-0bf7-4462-a62b-2aee12c53d78","Type":"ContainerStarted","Data":"2ea37f609415e42e51b4568abf9a92464a83ecaaa48a35c5f14b0db8cd1d3613"}
Jul 01 16:58:53.189740 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.189724    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 16:58:53.190034 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.190012    2556 generic.go:356] "Generic (PLEG): container finished" podID="067f479a-c3e1-4a2e-badc-f863f901a14b" containerID="3d21d6db98e4ff66c33e07e635faa951cf70c66c63cc27ae2bb288afe09b4039" exitCode=1
Jul 01 16:58:53.190129 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.190076    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"1781d1b8c113ef3778d383af71eb406b8b804a31e72d07b160199722c5b0d6ad"}
Jul 01 16:58:53.190180 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.190131    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"fd8a4b9471c1e631f15aa9dd939008b4398fede378b706c398be952a80726c8a"}
Jul 01 16:58:53.190180 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.190162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"4a9ef5fdc4792216397f4940ad16d4a8f1de457fed972f823aabac7fe82c6f0c"}
Jul 01 16:58:53.190242 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.190191    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"8c4497e6f8aab3a256d9aefdb160eb5e86538246b7fb04c852f4cc22e95968c1"}
Jul 01 16:58:53.190242 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.190227    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerDied","Data":"3d21d6db98e4ff66c33e07e635faa951cf70c66c63cc27ae2bb288afe09b4039"}
Jul 01 16:58:53.191360 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.191330    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ddgms" event={"ID":"b8e605ac-da6b-40ed-8fa6-079d387ea570","Type":"ContainerStarted","Data":"7a6bcbbc0743f195da6800e76cf3549006b2cafbfe936431773fc81d14bd0dc3"}
Jul 01 16:58:53.192648 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.192492    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hq7vh" event={"ID":"b277c8b7-1860-4ca6-97db-9e64e6f2ec9d","Type":"ContainerStarted","Data":"861ed4a8db09a5a0bfe608c6f31bf1858c163b60e0d3b0095592924f88875034"}
Jul 01 16:58:53.196855 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.196834    2556 generic.go:356] "Generic (PLEG): container finished" podID="c53c6577-4cf4-45a2-802a-690ea2560a57" containerID="cb76f8bad054a6c8b23edbfa88893105914375953b2e65fd063252f68e6aae96" exitCode=0
Jul 01 16:58:53.196939 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.196902    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerDied","Data":"cb76f8bad054a6c8b23edbfa88893105914375953b2e65fd063252f68e6aae96"}
Jul 01 16:58:53.215831 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.215790    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f6pw8" podStartSLOduration=4.23089012 podStartE2EDuration="23.215778926s" totalImagesPullingTime="18.984888806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:53.215690412 +0000 UTC m=+23.558031072" watchObservedRunningTime="2026-07-01 16:58:53.215778926 +0000 UTC m=+23.558119598"
Jul 01 16:58:53.268906 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.268872    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ddgms" podStartSLOduration=4.294075536 podStartE2EDuration="23.268857908s" totalImagesPullingTime="18.974782372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:53.238034719 +0000 UTC m=+23.580375380" watchObservedRunningTime="2026-07-01 16:58:53.268857908 +0000 UTC m=+23.611198567"
Jul 01 16:58:53.606960 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:53.606812    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 16:58:54.067991 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.067609    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T16:58:53.606949095Z","UUID":"33a86c0b-c76d-454c-97d2-f47742587640","Handler":null,"Name":"","Endpoint":""}
Jul 01 16:58:54.067991 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.067799    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:54.067991 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:54.067905    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:54.068316 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.068267    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:54.068468 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:54.068444    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:54.069484 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.069465    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 16:58:54.069606 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.069509    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 16:58:54.201121 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.201080    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6phpp" event={"ID":"f18c7204-9546-4e24-ae93-4edb51af02dd","Type":"ContainerStarted","Data":"d2341cc4cc200b0d24418d679b76fdb17e14e6825f5c06a6a2fde74d70434937"}
Jul 01 16:58:54.203477 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.203404    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h" event={"ID":"8e950e64-9263-493a-8a07-39043d185975","Type":"ContainerStarted","Data":"41364bfe5d0a402fe787ed4686d9e8b733f9d00f069c0f85d590ac9ede9856f2"}
Jul 01 16:58:54.226623 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.226576    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hq7vh" podStartSLOduration=5.247687432 podStartE2EDuration="24.226554921s" totalImagesPullingTime="18.978867489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:53.292226627 +0000 UTC m=+23.634567287" watchObservedRunningTime="2026-07-01 16:58:54.226554921 +0000 UTC m=+24.568895563"
Jul 01 16:58:54.227134 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:54.227097    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6phpp" podStartSLOduration=5.253321582 podStartE2EDuration="24.227086067s" totalImagesPullingTime="18.973764485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:54.227074671 +0000 UTC m=+24.569415329" watchObservedRunningTime="2026-07-01 16:58:54.227086067 +0000 UTC m=+24.569426729"
Jul 01 16:58:55.067968 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:55.067937    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:55.068144 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:55.068086    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:55.209684 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:55.209597    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h" event={"ID":"8e950e64-9263-493a-8a07-39043d185975","Type":"ContainerStarted","Data":"b8a6202d92b9cfbc286ebc027b46f1ec68a1f98519330c516f921a8f75e01bba"}
Jul 01 16:58:55.212876 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:55.212856    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 16:58:55.213462 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:55.213408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"19f9edf7c6cefead4f0e8ff74e0c7ce0a8064fab6d3cddfa3533fb097f0a714f"}
Jul 01 16:58:55.232193 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:55.232140    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gmv4h" podStartSLOduration=3.792475675 podStartE2EDuration="25.232123985s" totalImagesPullingTime="21.43964831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:55.232087429 +0000 UTC m=+25.574428082" watchObservedRunningTime="2026-07-01 16:58:55.232123985 +0000 UTC m=+25.574464636"
Jul 01 16:58:56.068459 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:56.068424    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:56.068605 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:56.068543    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:56.068605 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:56.068598    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:56.068715 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:56.068696    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:56.096184 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:56.096154    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:56.096750 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:56.096732    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:56.215667 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:56.215639    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:56.216268 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:56.215908    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ddgms"
Jul 01 16:58:57.068814 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:57.068672    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:57.068916 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:57.068896    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:57.222030 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:57.221998    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 16:58:57.222649 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:57.222514    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"4941892c49868523447efb840171f9a81dbccbadd66dc79629e118f941fd6710"}
Jul 01 16:58:57.223121 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:57.223104    2556 scope.go:117] "RemoveContainer" containerID="3d21d6db98e4ff66c33e07e635faa951cf70c66c63cc27ae2bb288afe09b4039"
Jul 01 16:58:58.068176 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.068140    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:58.068176 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.068156    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:58.068453 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:58.068262    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:58.068453 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:58.068397    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:58:58.228819 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.228795    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 16:58:58.229306 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.229259    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" event={"ID":"067f479a-c3e1-4a2e-badc-f863f901a14b","Type":"ContainerStarted","Data":"72ac2103b90cd8f2af54c7d2f618fa79faba78204dda6a078867007725ffad33"}
Jul 01 16:58:58.229566 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.229544    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:58.229670 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.229622    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:58.229670 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.229657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:58.249447 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.249422    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:58.249620 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.249560    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:58:58.275461 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:58.275380    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l" podStartSLOduration=9.222496122 podStartE2EDuration="28.27535997s" totalImagesPullingTime="19.052863848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:58:58.274841267 +0000 UTC m=+28.617181927" watchObservedRunningTime="2026-07-01 16:58:58.27535997 +0000 UTC m=+28.617700631"
Jul 01 16:58:59.067875 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:59.067849    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:59.068015 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:59.067945    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:59.706167 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:59.705991    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-chkmv"]
Jul 01 16:58:59.706549 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:59.706274    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:58:59.706549 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:59.706388    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:58:59.707310 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:59.707287    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rkmmg"]
Jul 01 16:58:59.707396 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:59.707369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:58:59.707463 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:59.707440    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:58:59.727153 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:59.727105    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-npk7n"]
Jul 01 16:58:59.727250 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:58:59.727188    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:58:59.727288 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:58:59.727268    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:59:00.235413 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:00.235379    2556 generic.go:356] "Generic (PLEG): container finished" podID="c53c6577-4cf4-45a2-802a-690ea2560a57" containerID="2cfb50f6cc82e356d06d1305a08bb254e4d97e1b831d52ab0b68b172bdd5f51f" exitCode=0
Jul 01 16:59:00.235597 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:00.235470    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerDied","Data":"2cfb50f6cc82e356d06d1305a08bb254e4d97e1b831d52ab0b68b172bdd5f51f"}
Jul 01 16:59:01.068119 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:01.068030    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:01.068759 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:01.068136    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:59:01.068759 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:01.068252    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:59:01.068759 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:01.068288    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:59:01.068759 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:01.068395    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:59:01.068759 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:01.068501    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:59:02.241742 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:02.241707    2556 generic.go:356] "Generic (PLEG): container finished" podID="c53c6577-4cf4-45a2-802a-690ea2560a57" containerID="fc2167f7929f8d8f592095b0dd0f49fa41345d69142b76656af5c77f88fc7180" exitCode=0
Jul 01 16:59:02.242330 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:02.241764    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerDied","Data":"fc2167f7929f8d8f592095b0dd0f49fa41345d69142b76656af5c77f88fc7180"}
Jul 01 16:59:03.067973 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:03.067938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:59:03.067973 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:03.067976    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:03.068216 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:03.067985    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:59:03.068216 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.068077    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-npk7n" podUID="40127afb-5366-4e79-9f41-64186c92e106"
Jul 01 16:59:03.068216 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.068168    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rkmmg" podUID="0f425d31-4437-4e8d-bd20-3c3e40bdb24b"
Jul 01 16:59:03.068312 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.068232    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chkmv" podUID="406c4f26-829c-4bbc-b670-2adde865c36f"
Jul 01 16:59:03.857633 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:03.857428    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:59:03.858131 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:03.857642    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:59:03.858131 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.857573    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 16:59:03.858131 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.857722    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:59:03.858131 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.857766    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret podName:40127afb-5366-4e79-9f41-64186c92e106 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:35.857745545 +0000 UTC m=+66.200086188 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret") pod "global-pull-secret-syncer-npk7n" (UID: "40127afb-5366-4e79-9f41-64186c92e106") : object "kube-system"/"original-pull-secret" not registered
Jul 01 16:59:03.858131 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.857793    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs podName:406c4f26-829c-4bbc-b670-2adde865c36f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:35.85778014 +0000 UTC m=+66.200120785 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs") pod "network-metrics-daemon-chkmv" (UID: "406c4f26-829c-4bbc-b670-2adde865c36f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 16:59:03.957943 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:03.957900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:03.958132 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.958057    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 16:59:03.958132 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.958086    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 16:59:03.958132 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.958096    2556 projected.go:196] Error preparing data for projected volume kube-api-access-bw22c for pod openshift-network-diagnostics/network-check-target-rkmmg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:59:03.958235 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:03.958143    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c podName:0f425d31-4437-4e8d-bd20-3c3e40bdb24b nodeName:}" failed. No retries permitted until 2026-07-01 16:59:35.95813026 +0000 UTC m=+66.300470898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bw22c" (UniqueName: "kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c") pod "network-check-target-rkmmg" (UID: "0f425d31-4437-4e8d-bd20-3c3e40bdb24b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 16:59:04.249240 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:04.249211    2556 generic.go:356] "Generic (PLEG): container finished" podID="c53c6577-4cf4-45a2-802a-690ea2560a57" containerID="4106928bd8c429f3523e9d4ea9e6b18cf589446f4ee1ee90e87774422b03eb70" exitCode=0
Jul 01 16:59:04.249397 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:04.249271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerDied","Data":"4106928bd8c429f3523e9d4ea9e6b18cf589446f4ee1ee90e87774422b03eb70"}
Jul 01 16:59:04.957743 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:04.957642    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-111.ec2.internal" event="NodeReady"
Jul 01 16:59:04.958132 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:04.957789    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 16:59:05.014780 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.014743    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vx286"]
Jul 01 16:59:05.018429 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.018401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.019224 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.019198    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h6jjk"]
Jul 01 16:59:05.021786 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.021762    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 16:59:05.022026 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.022011    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-shlnm\""
Jul 01 16:59:05.022205 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.022183    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 16:59:05.023020 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.022999    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.025508 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.025478    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9cnfg\""
Jul 01 16:59:05.025612 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.025517    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 16:59:05.025796 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.025775    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 16:59:05.025887 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.025778    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 16:59:05.033404 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.033379    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vx286"]
Jul 01 16:59:05.035678 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.035657    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6jjk"]
Jul 01 16:59:05.068550 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.068523    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:59:05.068550 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.068558    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:05.068761 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.068535    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:59:05.071602 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.071567    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 16:59:05.071735 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.071606    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 16:59:05.071910 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.071891    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8njwh\""
Jul 01 16:59:05.071910 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.071902    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sw8bs\""
Jul 01 16:59:05.072097 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.071955    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 16:59:05.072097 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.071998    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 16:59:05.168046 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.167999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.168244 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.168081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxh5\" (UniqueName: \"kubernetes.io/projected/af5c8513-b418-4256-8a57-6cf1381096b9-kube-api-access-qhxh5\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.168244 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.168141    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5c8513-b418-4256-8a57-6cf1381096b9-tmp-dir\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.168244 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.168167    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/af5c8513-b418-4256-8a57-6cf1381096b9-config-volume\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.168244 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.168195    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6f8q\" (UniqueName: \"kubernetes.io/projected/b8e0b3b7-2505-4bdb-bf4f-285c943fece3-kube-api-access-r6f8q\") pod \"ingress-canary-h6jjk\" (UID: \"b8e0b3b7-2505-4bdb-bf4f-285c943fece3\") " pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.168244 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.168243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8e0b3b7-2505-4bdb-bf4f-285c943fece3-cert\") pod \"ingress-canary-h6jjk\" (UID: \"b8e0b3b7-2505-4bdb-bf4f-285c943fece3\") " pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.269172 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269144    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5c8513-b418-4256-8a57-6cf1381096b9-tmp-dir\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.269347 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269177    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/af5c8513-b418-4256-8a57-6cf1381096b9-config-volume\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.269347 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6f8q\" (UniqueName: \"kubernetes.io/projected/b8e0b3b7-2505-4bdb-bf4f-285c943fece3-kube-api-access-r6f8q\") pod \"ingress-canary-h6jjk\" (UID: \"b8e0b3b7-2505-4bdb-bf4f-285c943fece3\") " pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.269347 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269237    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8e0b3b7-2505-4bdb-bf4f-285c943fece3-cert\") pod \"ingress-canary-h6jjk\" (UID: \"b8e0b3b7-2505-4bdb-bf4f-285c943fece3\") " pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.269558 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269527    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.269669 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxh5\" (UniqueName: \"kubernetes.io/projected/af5c8513-b418-4256-8a57-6cf1381096b9-kube-api-access-qhxh5\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.269669 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269536    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af5c8513-b418-4256-8a57-6cf1381096b9-tmp-dir\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.269669 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.269610    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:05.269669 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.269667    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls podName:af5c8513-b418-4256-8a57-6cf1381096b9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:05.769649917 +0000 UTC m=+36.111990558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls") pod "dns-default-vx286" (UID: "af5c8513-b418-4256-8a57-6cf1381096b9") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:05.269878 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.269857    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/af5c8513-b418-4256-8a57-6cf1381096b9-config-volume\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.274146 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.274124    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b8e0b3b7-2505-4bdb-bf4f-285c943fece3-cert\") pod \"ingress-canary-h6jjk\" (UID: \"b8e0b3b7-2505-4bdb-bf4f-285c943fece3\") " pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.303518 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.303486    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6f8q\" (UniqueName: \"kubernetes.io/projected/b8e0b3b7-2505-4bdb-bf4f-285c943fece3-kube-api-access-r6f8q\") pod \"ingress-canary-h6jjk\" (UID: \"b8e0b3b7-2505-4bdb-bf4f-285c943fece3\") " pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.317577 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.317547    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"]
Jul 01 16:59:05.323143 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.323124    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"
Jul 01 16:59:05.327017 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.326997    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b9ssz\""
Jul 01 16:59:05.327402 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.327384    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 01 16:59:05.329824 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.329804    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 01 16:59:05.339166 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.339146    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6jjk"
Jul 01 16:59:05.340121 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.340101    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"]
Jul 01 16:59:05.350949 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.350928    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxh5\" (UniqueName: \"kubernetes.io/projected/af5c8513-b418-4256-8a57-6cf1381096b9-kube-api-access-qhxh5\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.470680 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.470576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kkd\" (UniqueName: \"kubernetes.io/projected/b4c671d8-43bd-4a42-a885-81d70371a7d3-kube-api-access-q9kkd\") pod \"volume-data-source-validator-69d9f679db-hddrd\" (UID: \"b4c671d8-43bd-4a42-a885-81d70371a7d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"
Jul 01 16:59:05.505906 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.505876    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"]
Jul 01 16:59:05.510945 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.510920    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.519387 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.519210    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79bb6d4bff-plxzt"]
Jul 01 16:59:05.521238 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.521169    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 01 16:59:05.522668 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.522642    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 16:59:05.522668 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.522667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.522842 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.522681    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 16:59:05.522842 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.522697    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hdhz4\""
Jul 01 16:59:05.522842 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.522753    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 01 16:59:05.525031 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.524981    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"]
Jul 01 16:59:05.527292 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.527130    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 01 16:59:05.527292 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.527212    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 01 16:59:05.527292 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.527219    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 01 16:59:05.527292 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.527271    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 01 16:59:05.528338 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.527485    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xttn7\""
Jul 01 16:59:05.528338 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.527500    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 01 16:59:05.528338 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.527758    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 01 16:59:05.532876 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.532855    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6jjk"]
Jul 01 16:59:05.536849 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:05.536824    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e0b3b7_2505_4bdb_bf4f_285c943fece3.slice/crio-b978be56b2dee84205a149f58f109a55fa81740780a7290c21c7802313c8667b WatchSource:0}: Error finding container b978be56b2dee84205a149f58f109a55fa81740780a7290c21c7802313c8667b: Status 404 returned error can't find the container with id b978be56b2dee84205a149f58f109a55fa81740780a7290c21c7802313c8667b
Jul 01 16:59:05.546199 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.546126    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79bb6d4bff-plxzt"]
Jul 01 16:59:05.571054 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.571026    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kkd\" (UniqueName: \"kubernetes.io/projected/b4c671d8-43bd-4a42-a885-81d70371a7d3-kube-api-access-q9kkd\") pod \"volume-data-source-validator-69d9f679db-hddrd\" (UID: \"b4c671d8-43bd-4a42-a885-81d70371a7d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"
Jul 01 16:59:05.583302 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.583279    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kkd\" (UniqueName: \"kubernetes.io/projected/b4c671d8-43bd-4a42-a885-81d70371a7d3-kube-api-access-q9kkd\") pod \"volume-data-source-validator-69d9f679db-hddrd\" (UID: \"b4c671d8-43bd-4a42-a885-81d70371a7d3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"
Jul 01 16:59:05.633782 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.633750    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"
Jul 01 16:59:05.671606 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671575    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-default-certificate\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.671762 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mbtk\" (UniqueName: \"kubernetes.io/projected/6dd03f61-baea-420b-bf09-060e6338f8f4-kube-api-access-2mbtk\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.671762 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671689    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.671762 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgtw\" (UniqueName: \"kubernetes.io/projected/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-kube-api-access-kjgtw\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.671889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.671889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671803    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-stats-auth\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.671889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671840    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.671889 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.671871    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.772592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.772513    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-default-certificate\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.772592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.772565    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:05.772807 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.772601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbtk\" (UniqueName: \"kubernetes.io/projected/6dd03f61-baea-420b-bf09-060e6338f8f4-kube-api-access-2mbtk\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.772807 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.772699    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:05.772807 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.772760    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls podName:af5c8513-b418-4256-8a57-6cf1381096b9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:06.772739926 +0000 UTC m=+37.115080577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls") pod "dns-default-vx286" (UID: "af5c8513-b418-4256-8a57-6cf1381096b9") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:05.772967 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.772916    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.773023 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.772965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgtw\" (UniqueName: \"kubernetes.io/projected/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-kube-api-access-kjgtw\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.773023 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.773002    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.773144 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.773026    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-stats-auth\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.773144 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.773092    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:06.273073346 +0000 UTC m=+36.615414007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : configmap references non-existent config key: service-ca.crt
Jul 01 16:59:05.773144 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.773116    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.773295 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.773145    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.773295 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.773242    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 16:59:05.773295 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.773291    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:06.273276925 +0000 UTC m=+36.615617577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : secret "router-metrics-certs-default" not found
Jul 01 16:59:05.773439 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.773301    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:05.773439 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:05.773347    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls podName:fb1e46ce-6d0c-4bcd-b895-01f74c6a829f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:06.273333075 +0000 UTC m=+36.615673723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-44p68" (UID: "fb1e46ce-6d0c-4bcd-b895-01f74c6a829f") : secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:05.773772 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.773746    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-telemetry-config\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.775674 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.775654    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-default-certificate\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.775860 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.775841    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-stats-auth\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:05.778185 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.778164    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd"]
Jul 01 16:59:05.781723 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:05.781701    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c671d8_43bd_4a42_a885_81d70371a7d3.slice/crio-23117e6665b30160b94a48058d4583f515b1b41e37051b46d67eb38e63b8ef9a WatchSource:0}: Error finding container 23117e6665b30160b94a48058d4583f515b1b41e37051b46d67eb38e63b8ef9a: Status 404 returned error can't find the container with id 23117e6665b30160b94a48058d4583f515b1b41e37051b46d67eb38e63b8ef9a
Jul 01 16:59:05.786265 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.786243    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgtw\" (UniqueName: \"kubernetes.io/projected/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-kube-api-access-kjgtw\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:05.786265 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:05.786256    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbtk\" (UniqueName: \"kubernetes.io/projected/6dd03f61-baea-420b-bf09-060e6338f8f4-kube-api-access-2mbtk\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:06.257573 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:06.257468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd" event={"ID":"b4c671d8-43bd-4a42-a885-81d70371a7d3","Type":"ContainerStarted","Data":"23117e6665b30160b94a48058d4583f515b1b41e37051b46d67eb38e63b8ef9a"}
Jul 01 16:59:06.259471 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:06.259409    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6jjk" event={"ID":"b8e0b3b7-2505-4bdb-bf4f-285c943fece3","Type":"ContainerStarted","Data":"b978be56b2dee84205a149f58f109a55fa81740780a7290c21c7802313c8667b"}
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:06.278112    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:06.278412    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:06.278532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:06.278688    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:07.278669367 +0000 UTC m=+37.621010020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : configmap references non-existent config key: service-ca.crt
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:06.279051    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:06.279122    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:07.279104644 +0000 UTC m=+37.621445295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : secret "router-metrics-certs-default" not found
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:06.279253    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:06.279325 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:06.279302    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls podName:fb1e46ce-6d0c-4bcd-b895-01f74c6a829f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:07.279289831 +0000 UTC m=+37.621630472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-44p68" (UID: "fb1e46ce-6d0c-4bcd-b895-01f74c6a829f") : secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:06.782800 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:06.782761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:06.782992 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:06.782912    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:06.782992 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:06.782972    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls podName:af5c8513-b418-4256-8a57-6cf1381096b9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:08.782955608 +0000 UTC m=+39.125296268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls") pod "dns-default-vx286" (UID: "af5c8513-b418-4256-8a57-6cf1381096b9") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:07.287001 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:07.286961    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:07.287594 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:07.287034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:07.287594 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:07.287078    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:07.287594 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:07.287181    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:07.287594 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:07.287204    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:09.287185648 +0000 UTC m=+39.629526293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : configmap references non-existent config key: service-ca.crt
Jul 01 16:59:07.287594 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:07.287244    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls podName:fb1e46ce-6d0c-4bcd-b895-01f74c6a829f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:09.287227212 +0000 UTC m=+39.629567853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-44p68" (UID: "fb1e46ce-6d0c-4bcd-b895-01f74c6a829f") : secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:07.287594 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:07.287293    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 16:59:07.287594 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:07.287349    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:09.287333087 +0000 UTC m=+39.629673726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : secret "router-metrics-certs-default" not found
Jul 01 16:59:08.799992 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:08.799929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:08.800473 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:08.800113    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:08.800473 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:08.800182    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls podName:af5c8513-b418-4256-8a57-6cf1381096b9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:12.800166131 +0000 UTC m=+43.142506785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls") pod "dns-default-vx286" (UID: "af5c8513-b418-4256-8a57-6cf1381096b9") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:09.304592 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:09.304558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:09.304767 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:09.304603    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:09.304767 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:09.304721    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 16:59:09.304880 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:09.304722    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:09.304927 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:09.304813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:09.304927 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:09.304888    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls podName:fb1e46ce-6d0c-4bcd-b895-01f74c6a829f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:13.304865191 +0000 UTC m=+43.647205845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-44p68" (UID: "fb1e46ce-6d0c-4bcd-b895-01f74c6a829f") : secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:09.304927 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:09.304920    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:13.304905285 +0000 UTC m=+43.647245924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : configmap references non-existent config key: service-ca.crt
Jul 01 16:59:09.305098 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:09.304940    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:13.304930234 +0000 UTC m=+43.647270872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : secret "router-metrics-certs-default" not found
Jul 01 16:59:10.275876 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:10.275836    2556 generic.go:356] "Generic (PLEG): container finished" podID="c53c6577-4cf4-45a2-802a-690ea2560a57" containerID="498fa77ce171d9b48b257ab99dc1dc071e16432084b3a7c88122b70352fe3d35" exitCode=0
Jul 01 16:59:10.276753 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:10.275912    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerDied","Data":"498fa77ce171d9b48b257ab99dc1dc071e16432084b3a7c88122b70352fe3d35"}
Jul 01 16:59:10.277679 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:10.277654    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd" event={"ID":"b4c671d8-43bd-4a42-a885-81d70371a7d3","Type":"ContainerStarted","Data":"4bd50abbd0d626fe134ec0a7b3f47f3bd80233dda5d5a4277514ba6ee5c5c640"}
Jul 01 16:59:10.281512 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:10.281477    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6jjk" event={"ID":"b8e0b3b7-2505-4bdb-bf4f-285c943fece3","Type":"ContainerStarted","Data":"135522af2cfc0244430c32deb540d2dc8188fae03d6e7f858dde89d2bec30648"}
Jul 01 16:59:10.327947 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:10.327898    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h6jjk" podStartSLOduration=1.9151795969999998 podStartE2EDuration="6.327881527s" totalImagesPullingTime="4.41270193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:10.326606873 +0000 UTC m=+40.668947533" watchObservedRunningTime="2026-07-01 16:59:10.327881527 +0000 UTC m=+40.670222187"
Jul 01 16:59:10.350456 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:10.350169    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-69d9f679db-hddrd" podStartSLOduration=1.221089253 podStartE2EDuration="5.350144902s" totalImagesPullingTime="4.129055649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:10.348845791 +0000 UTC m=+40.691186453" watchObservedRunningTime="2026-07-01 16:59:10.350144902 +0000 UTC m=+40.692485577"
Jul 01 16:59:11.145653 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.145625    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5bl5g"]
Jul 01 16:59:11.164246 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.164220    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bl5g"]
Jul 01 16:59:11.164376 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.164338    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.167976 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.167953    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 16:59:11.168139 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.167953    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 16:59:11.168139 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.168055    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 16:59:11.168139 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.168108    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zdcr6\""
Jul 01 16:59:11.168372 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.168356    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 16:59:11.286750 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.286674    2556 generic.go:356] "Generic (PLEG): container finished" podID="c53c6577-4cf4-45a2-802a-690ea2560a57" containerID="6ce3b2f645d4ae0ec14772085e35375d327a4ad1949c9a01735f666b7ac0e60b" exitCode=0
Jul 01 16:59:11.287197 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.286760    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerDied","Data":"6ce3b2f645d4ae0ec14772085e35375d327a4ad1949c9a01735f666b7ac0e60b"}
Jul 01 16:59:11.321206 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.321179    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.321298 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.321260    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-crio-socket\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.321385 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.321367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-data-volume\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.321452 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.321409    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.321512 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.321449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phw2c\" (UniqueName: \"kubernetes.io/projected/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-api-access-phw2c\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.422344 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.422306    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-data-volume\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.422495 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.422360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.422495 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.422391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phw2c\" (UniqueName: \"kubernetes.io/projected/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-api-access-phw2c\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.422495 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.422440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.422639 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.422500    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:11.422639 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.422534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-crio-socket\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.422639 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.422570    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:11.922550797 +0000 UTC m=+42.264891436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:11.422639 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.422617    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-data-volume\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.422827 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.422505    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:11.422827 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.422691    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:11.922673493 +0000 UTC m=+42.265014149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:11.422827 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.422699    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-crio-socket\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.443245 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.443213    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phw2c\" (UniqueName: \"kubernetes.io/projected/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-api-access-phw2c\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.925607 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.925570    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.925788 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:11.925626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:11.925788 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.925700    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:11.925788 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.925726    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:11.925788 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.925755    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:12.925742352 +0000 UTC m=+43.268082990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:11.925788 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:11.925780    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:12.925767664 +0000 UTC m=+43.268108305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:12.291804 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:12.291767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" event={"ID":"c53c6577-4cf4-45a2-802a-690ea2560a57","Type":"ContainerStarted","Data":"bea70ced6ca393f23618c6a7438109da1995e9ca334dbab3661d60bacea210a0"}
Jul 01 16:59:12.382638 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:12.382585    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fwnqp" podStartSLOduration=9.407808606 podStartE2EDuration="42.38256857s" totalImagesPullingTime="32.974759964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:12.382541238 +0000 UTC m=+42.724881902" watchObservedRunningTime="2026-07-01 16:59:12.38256857 +0000 UTC m=+42.724909224"
Jul 01 16:59:12.832185 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:12.832146    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:12.832365 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:12.832257    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:12.832365 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:12.832309    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls podName:af5c8513-b418-4256-8a57-6cf1381096b9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:20.832296481 +0000 UTC m=+51.174637118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls") pod "dns-default-vx286" (UID: "af5c8513-b418-4256-8a57-6cf1381096b9") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:12.932526 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:12.932490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:12.932664 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:12.932544    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:12.932664 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:12.932645    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:12.932732 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:12.932693    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:14.932679727 +0000 UTC m=+45.275020368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:12.932772 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:12.932644    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:12.932801 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:12.932782    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:14.932768009 +0000 UTC m=+45.275108647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:13.334971 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:13.334936    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:13.335426 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:13.334982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:13.335426 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:13.335008    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:13.335426 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:13.335136    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:13.335426 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:13.335151    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 16:59:13.335426 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:13.335219    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls podName:fb1e46ce-6d0c-4bcd-b895-01f74c6a829f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:21.335198743 +0000 UTC m=+51.677539385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-44p68" (UID: "fb1e46ce-6d0c-4bcd-b895-01f74c6a829f") : secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:13.335426 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:13.335241    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:21.335231845 +0000 UTC m=+51.677572486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : secret "router-metrics-certs-default" not found
Jul 01 16:59:13.335426 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:13.335258    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:21.335250266 +0000 UTC m=+51.677590904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : configmap references non-existent config key: service-ca.crt
Jul 01 16:59:14.030651 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:14.030626    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hq7vh_b277c8b7-1860-4ca6-97db-9e64e6f2ec9d/dns-node-resolver/0.log"
Jul 01 16:59:14.744782 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:14.744754    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f6pw8_1f0b617c-0bf7-4462-a62b-2aee12c53d78/node-ca/0.log"
Jul 01 16:59:14.947507 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:14.947471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:14.947666 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:14.947523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:14.947666 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:14.947585    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:14.947666 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:14.947630    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:18.94761789 +0000 UTC m=+49.289958528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:14.947666 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:14.947633    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:14.947807 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:14.947685    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:18.947669289 +0000 UTC m=+49.290009927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:15.746492 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:15.746464    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6jjk_b8e0b3b7-2505-4bdb-bf4f-285c943fece3/serve-healthcheck-canary/0.log"
Jul 01 16:59:18.979934 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:18.979892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:18.980424 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:18.979948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:18.980424 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:18.980054    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:18.980424 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:18.980112    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 16:59:18.980424 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:18.980156    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:26.980136271 +0000 UTC m=+57.322476910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:18.980424 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:18.980176    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:26.980166388 +0000 UTC m=+57.322507041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : configmap "kube-rbac-proxy" not found
Jul 01 16:59:20.893668 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:20.893628    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:20.894044 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:20.893780    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 16:59:20.894044 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:20.893849    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls podName:af5c8513-b418-4256-8a57-6cf1381096b9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:36.893832722 +0000 UTC m=+67.236173359 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls") pod "dns-default-vx286" (UID: "af5c8513-b418-4256-8a57-6cf1381096b9") : secret "dns-default-metrics-tls" not found
Jul 01 16:59:21.397881 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:21.397841    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:21.398113 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:21.397898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:21.398113 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:21.397946    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:21.398113 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:21.398010    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:37.397988137 +0000 UTC m=+67.740328795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : configmap references non-existent config key: service-ca.crt
Jul 01 16:59:21.398113 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:21.398056    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:21.398113 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:21.398087    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 01 16:59:21.398113 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:21.398117    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls podName:fb1e46ce-6d0c-4bcd-b895-01f74c6a829f nodeName:}" failed. No retries permitted until 2026-07-01 16:59:37.398106814 +0000 UTC m=+67.740447455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-54cb84584-44p68" (UID: "fb1e46ce-6d0c-4bcd-b895-01f74c6a829f") : secret "cluster-monitoring-operator-tls" not found
Jul 01 16:59:21.398343 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:21.398137    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs podName:6dd03f61-baea-420b-bf09-060e6338f8f4 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:37.398122381 +0000 UTC m=+67.740463019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs") pod "router-default-79bb6d4bff-plxzt" (UID: "6dd03f61-baea-420b-bf09-060e6338f8f4") : secret "router-metrics-certs-default" not found
Jul 01 16:59:27.040656 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:27.040621    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:27.041103 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:27.040674    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:27.041103 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:27.040770    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:27.041103 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:27.040834    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls podName:c9712d61-9858-4ec0-ae38-e3e16e9fa2d9 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:43.040819791 +0000 UTC m=+73.383160429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bl5g" (UID: "c9712d61-9858-4ec0-ae38-e3e16e9fa2d9") : secret "insights-runtime-extractor-tls" not found
Jul 01 16:59:27.041213 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:27.041171    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:30.245843 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:30.245815    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cht4l"
Jul 01 16:59:35.912470 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.912414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:59:35.912470 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.912473    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:59:35.916504 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.916486    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 16:59:35.916550 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.916487    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 16:59:35.924879 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.924850    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/406c4f26-829c-4bbc-b670-2adde865c36f-metrics-certs\") pod \"network-metrics-daemon-chkmv\" (UID: \"406c4f26-829c-4bbc-b670-2adde865c36f\") " pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:59:35.924993 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.924900    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/40127afb-5366-4e79-9f41-64186c92e106-original-pull-secret\") pod \"global-pull-secret-syncer-npk7n\" (UID: \"40127afb-5366-4e79-9f41-64186c92e106\") " pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:59:35.982150 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.982116    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8njwh\""
Jul 01 16:59:35.985869 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.985852    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-npk7n"
Jul 01 16:59:35.989868 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:35.989845    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chkmv"
Jul 01 16:59:36.012968 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.012940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:36.016024 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.015998    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 16:59:36.030255 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.029987    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 16:59:36.036332 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.036308    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw22c\" (UniqueName: \"kubernetes.io/projected/0f425d31-4437-4e8d-bd20-3c3e40bdb24b-kube-api-access-bw22c\") pod \"network-check-target-rkmmg\" (UID: \"0f425d31-4437-4e8d-bd20-3c3e40bdb24b\") " pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:36.124095 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.123939    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-npk7n"]
Jul 01 16:59:36.126561 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:36.126534    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40127afb_5366_4e79_9f41_64186c92e106.slice/crio-51110c915fe5fe74bf9faeacb3f81e220795997519df34b16ee855d4f0986f89 WatchSource:0}: Error finding container 51110c915fe5fe74bf9faeacb3f81e220795997519df34b16ee855d4f0986f89: Status 404 returned error can't find the container with id 51110c915fe5fe74bf9faeacb3f81e220795997519df34b16ee855d4f0986f89
Jul 01 16:59:36.142351 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.142329    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-chkmv"]
Jul 01 16:59:36.145007 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:36.144984    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406c4f26_829c_4bbc_b670_2adde865c36f.slice/crio-b02521610d85d024c27905f90257b4562886fd2e64f08497c59d9179663a16e4 WatchSource:0}: Error finding container b02521610d85d024c27905f90257b4562886fd2e64f08497c59d9179663a16e4: Status 404 returned error can't find the container with id b02521610d85d024c27905f90257b4562886fd2e64f08497c59d9179663a16e4
Jul 01 16:59:36.295535 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.295505    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sw8bs\""
Jul 01 16:59:36.303284 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.303253    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:36.350046 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.350001    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chkmv" event={"ID":"406c4f26-829c-4bbc-b670-2adde865c36f","Type":"ContainerStarted","Data":"b02521610d85d024c27905f90257b4562886fd2e64f08497c59d9179663a16e4"}
Jul 01 16:59:36.351587 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.351558    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-npk7n" event={"ID":"40127afb-5366-4e79-9f41-64186c92e106","Type":"ContainerStarted","Data":"51110c915fe5fe74bf9faeacb3f81e220795997519df34b16ee855d4f0986f89"}
Jul 01 16:59:36.424802 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.424774    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rkmmg"]
Jul 01 16:59:36.427874 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:36.427845    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f425d31_4437_4e8d_bd20_3c3e40bdb24b.slice/crio-04e742b7abf23deddf96dfb6f52c3965c65148da12bf282ce6a9d9ac8b271be1 WatchSource:0}: Error finding container 04e742b7abf23deddf96dfb6f52c3965c65148da12bf282ce6a9d9ac8b271be1: Status 404 returned error can't find the container with id 04e742b7abf23deddf96dfb6f52c3965c65148da12bf282ce6a9d9ac8b271be1
Jul 01 16:59:36.920204 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.920169    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:36.922993 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:36.922965    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/af5c8513-b418-4256-8a57-6cf1381096b9-metrics-tls\") pod \"dns-default-vx286\" (UID: \"af5c8513-b418-4256-8a57-6cf1381096b9\") " pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:37.137477 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.137443    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-shlnm\""
Jul 01 16:59:37.144881 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.144842    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:37.311743 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.311713    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vx286"]
Jul 01 16:59:37.314556 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:37.314522    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5c8513_b418_4256_8a57_6cf1381096b9.slice/crio-93e1044f5e17451b660d605a28d507c40f355ddeb870b37898e7c8467398a98b WatchSource:0}: Error finding container 93e1044f5e17451b660d605a28d507c40f355ddeb870b37898e7c8467398a98b: Status 404 returned error can't find the container with id 93e1044f5e17451b660d605a28d507c40f355ddeb870b37898e7c8467398a98b
Jul 01 16:59:37.356812 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.356735    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rkmmg" event={"ID":"0f425d31-4437-4e8d-bd20-3c3e40bdb24b","Type":"ContainerStarted","Data":"04e742b7abf23deddf96dfb6f52c3965c65148da12bf282ce6a9d9ac8b271be1"}
Jul 01 16:59:37.358136 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.358099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vx286" event={"ID":"af5c8513-b418-4256-8a57-6cf1381096b9","Type":"ContainerStarted","Data":"93e1044f5e17451b660d605a28d507c40f355ddeb870b37898e7c8467398a98b"}
Jul 01 16:59:37.424751 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.424713    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:37.424947 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.424783    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:37.424947 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.424809    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:37.425632 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.425603    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd03f61-baea-420b-bf09-060e6338f8f4-service-ca-bundle\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:37.427972 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.427949    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd03f61-baea-420b-bf09-060e6338f8f4-metrics-certs\") pod \"router-default-79bb6d4bff-plxzt\" (UID: \"6dd03f61-baea-420b-bf09-060e6338f8f4\") " pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:37.428686 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.428645    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb1e46ce-6d0c-4bcd-b895-01f74c6a829f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-54cb84584-44p68\" (UID: \"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f\") " pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:37.625123 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.625088    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hdhz4\""
Jul 01 16:59:37.632181 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.632149    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"
Jul 01 16:59:37.636560 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.636468    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xttn7\""
Jul 01 16:59:37.643702 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.643638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:37.809552 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.809529    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68"]
Jul 01 16:59:37.813291 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:37.813218    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1e46ce_6d0c_4bcd_b895_01f74c6a829f.slice/crio-80428139a6ae27923d0aa2c64e92c2f6eb353c86ba58f3fc817f3760fc0c8642 WatchSource:0}: Error finding container 80428139a6ae27923d0aa2c64e92c2f6eb353c86ba58f3fc817f3760fc0c8642: Status 404 returned error can't find the container with id 80428139a6ae27923d0aa2c64e92c2f6eb353c86ba58f3fc817f3760fc0c8642
Jul 01 16:59:37.828946 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:37.828910    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79bb6d4bff-plxzt"]
Jul 01 16:59:37.832707 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:37.832676    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd03f61_baea_420b_bf09_060e6338f8f4.slice/crio-72cc666d250b43c8d0e18a52d58bbcb6c3d25140d3c122e9168390c03692fe90 WatchSource:0}: Error finding container 72cc666d250b43c8d0e18a52d58bbcb6c3d25140d3c122e9168390c03692fe90: Status 404 returned error can't find the container with id 72cc666d250b43c8d0e18a52d58bbcb6c3d25140d3c122e9168390c03692fe90
Jul 01 16:59:38.364148 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:38.364111    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79bb6d4bff-plxzt" event={"ID":"6dd03f61-baea-420b-bf09-060e6338f8f4","Type":"ContainerStarted","Data":"22196e48356545fa69c3c24941a8aaaf31729b65249e8071f1d5af2a155ce38f"}
Jul 01 16:59:38.364698 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:38.364173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79bb6d4bff-plxzt" event={"ID":"6dd03f61-baea-420b-bf09-060e6338f8f4","Type":"ContainerStarted","Data":"72cc666d250b43c8d0e18a52d58bbcb6c3d25140d3c122e9168390c03692fe90"}
Jul 01 16:59:38.366076 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:38.366030    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68" event={"ID":"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f","Type":"ContainerStarted","Data":"80428139a6ae27923d0aa2c64e92c2f6eb353c86ba58f3fc817f3760fc0c8642"}
Jul 01 16:59:38.394375 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:38.394164    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-79bb6d4bff-plxzt" podStartSLOduration=33.394142058 podStartE2EDuration="33.394142058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:38.392964902 +0000 UTC m=+68.735305565" watchObservedRunningTime="2026-07-01 16:59:38.394142058 +0000 UTC m=+68.736482721"
Jul 01 16:59:38.644473 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:38.644292    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:38.647759 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:38.647558    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:39.369650 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:39.369616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:39.371607 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:39.371578    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79bb6d4bff-plxzt"
Jul 01 16:59:40.351305 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.351191    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"]
Jul 01 16:59:40.367446 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.367415    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.371721 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.371553    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"]
Jul 01 16:59:40.372141 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.372012    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 16:59:40.372374 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.372353    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 01 16:59:40.373779 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.373757    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 16:59:40.374018 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.374001    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 16:59:40.405920 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.405886    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"]
Jul 01 16:59:40.420678 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.420645    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.424158 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.424008    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 16:59:40.424158 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.424039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 16:59:40.424158 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.424039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 16:59:40.425973 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.425948    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 16:59:40.428361 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.428328    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"]
Jul 01 16:59:40.452091 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.451991    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0fbeebf3-0e81-432b-8620-34613fc5a5de-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.452264 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452203    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.452264 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452245    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-hub\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.452375 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452272    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.452375 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452308    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw78k\" (UniqueName: \"kubernetes.io/projected/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-kube-api-access-bw78k\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.452375 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452334    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rph4h\" (UniqueName: \"kubernetes.io/projected/0fbeebf3-0e81-432b-8620-34613fc5a5de-kube-api-access-rph4h\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.452522 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452411    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-ca\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.452522 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.452613 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.452527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbeebf3-0e81-432b-8620-34613fc5a5de-tmp\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.553346 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553313    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw78k\" (UniqueName: \"kubernetes.io/projected/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-kube-api-access-bw78k\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.553346 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rph4h\" (UniqueName: \"kubernetes.io/projected/0fbeebf3-0e81-432b-8620-34613fc5a5de-kube-api-access-rph4h\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.553593 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-ca\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.553593 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.553593 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553526    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbeebf3-0e81-432b-8620-34613fc5a5de-tmp\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.553593 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553582    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0fbeebf3-0e81-432b-8620-34613fc5a5de-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.553788 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553627    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.553788 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553731    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-hub\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.553788 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.553769    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.554116 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.554035    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fbeebf3-0e81-432b-8620-34613fc5a5de-tmp\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.554385 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.554360    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.556076 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.556041    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-ca\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.556228 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.556049    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0fbeebf3-0e81-432b-8620-34613fc5a5de-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.556228 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.556181    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.556372 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.556354    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-hub\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.556462 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.556444    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.575301 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.575274    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw78k\" (UniqueName: \"kubernetes.io/projected/cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a-kube-api-access-bw78k\") pod \"cluster-proxy-proxy-agent-d85f679d7-pz7s9\" (UID: \"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:40.584257 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.584233    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rph4h\" (UniqueName: \"kubernetes.io/projected/0fbeebf3-0e81-432b-8620-34613fc5a5de-kube-api-access-rph4h\") pod \"klusterlet-addon-workmgr-6d789c44f-2b9g9\" (UID: \"0fbeebf3-0e81-432b-8620-34613fc5a5de\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.682407 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.682326    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:40.743960 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:40.743920    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"
Jul 01 16:59:42.814577 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:42.814552    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9"]
Jul 01 16:59:42.823399 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:42.823339    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"]
Jul 01 16:59:43.075607 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.075574    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:43.078798 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.078766    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c9712d61-9858-4ec0-ae38-e3e16e9fa2d9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bl5g\" (UID: \"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9\") " pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:43.286556 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.286518    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zdcr6\""
Jul 01 16:59:43.293521 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.293492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bl5g"
Jul 01 16:59:43.408959 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.408917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9" event={"ID":"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a","Type":"ContainerStarted","Data":"efe8662e91ecd51fa4df7fc448e9deafb7a6ac8c90213c83e329b6573dd0dfc9"}
Jul 01 16:59:43.411352 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.411041    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chkmv" event={"ID":"406c4f26-829c-4bbc-b670-2adde865c36f","Type":"ContainerStarted","Data":"f384878bdbf20814ba50fc54c3338eb3b0512d0eb2f3c4b0a5e0533ca0c1d190"}
Jul 01 16:59:43.411352 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.411150    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chkmv" event={"ID":"406c4f26-829c-4bbc-b670-2adde865c36f","Type":"ContainerStarted","Data":"b27a384115790b37be5502716c7a4c22773e37e003e498aa19b3fd826e3a9cd7"}
Jul 01 16:59:43.412865 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.412841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-npk7n" event={"ID":"40127afb-5366-4e79-9f41-64186c92e106","Type":"ContainerStarted","Data":"21a893d1c68a34e6dcd0b44f14a532b73eeb3e6699a9fc679e76540fb56e5a73"}
Jul 01 16:59:43.414161 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.414134    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9" event={"ID":"0fbeebf3-0e81-432b-8620-34613fc5a5de","Type":"ContainerStarted","Data":"da8ebb70c4050df3163494adc318f6ba7e1fe56a99b867ab715b443dedd640e4"}
Jul 01 16:59:43.415617 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.415596    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rkmmg" event={"ID":"0f425d31-4437-4e8d-bd20-3c3e40bdb24b","Type":"ContainerStarted","Data":"97fffe88be1d6713791e320f0c8b2500c3d9c4056bc2efae24f706f743132cf7"}
Jul 01 16:59:43.415714 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.415666    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 16:59:43.417373 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.417354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vx286" event={"ID":"af5c8513-b418-4256-8a57-6cf1381096b9","Type":"ContainerStarted","Data":"40f430fe455a2cd9563c10cdd99c1ad887f83f59c0c6e713fb9c9747275129fb"}
Jul 01 16:59:43.417484 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.417396    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vx286" event={"ID":"af5c8513-b418-4256-8a57-6cf1381096b9","Type":"ContainerStarted","Data":"e4265612de474c3f5e2aacca13e523f37f8ef00d05341f1317d51e6cfe3478e1"}
Jul 01 16:59:43.417484 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.417444    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:43.418955 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.418922    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68" event={"ID":"fb1e46ce-6d0c-4bcd-b895-01f74c6a829f","Type":"ContainerStarted","Data":"ffcdcc1b10f98bacc101afa048a7271d5b82de388b8693be4df531199f965071"}
Jul 01 16:59:43.454120 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.454051    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-chkmv" podStartSLOduration=67.005163575 podStartE2EDuration="1m13.454036508s" totalImagesPullingTime="6.448872933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:43.453746254 +0000 UTC m=+73.796086914" watchObservedRunningTime="2026-07-01 16:59:43.454036508 +0000 UTC m=+73.796377168"
Jul 01 16:59:43.523260 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.523202    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vx286" podStartSLOduration=34.243344498 podStartE2EDuration="39.523175272s" totalImagesPullingTime="5.279830774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:43.51831216 +0000 UTC m=+73.860652820" watchObservedRunningTime="2026-07-01 16:59:43.523175272 +0000 UTC m=+73.865515963"
Jul 01 16:59:43.523421 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:43.523358    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9712d61_9858_4ec0_ae38_e3e16e9fa2d9.slice/crio-3a500b35e03ed55ac95a902307b860fd29d7f6323c9a90e16712d9394f5e846b WatchSource:0}: Error finding container 3a500b35e03ed55ac95a902307b860fd29d7f6323c9a90e16712d9394f5e846b: Status 404 returned error can't find the container with id 3a500b35e03ed55ac95a902307b860fd29d7f6323c9a90e16712d9394f5e846b
Jul 01 16:59:43.524784 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.524755    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bl5g"]
Jul 01 16:59:43.568982 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.568882    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-npk7n" podStartSLOduration=67.099861579 podStartE2EDuration="1m13.568865549s" totalImagesPullingTime="6.46900397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:43.567266426 +0000 UTC m=+73.909607086" watchObservedRunningTime="2026-07-01 16:59:43.568865549 +0000 UTC m=+73.911206209"
Jul 01 16:59:43.608547 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.608488    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rkmmg" podStartSLOduration=67.441227012 podStartE2EDuration="1m13.608467775s" totalImagesPullingTime="6.167240763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:43.607793669 +0000 UTC m=+73.950134331" watchObservedRunningTime="2026-07-01 16:59:43.608467775 +0000 UTC m=+73.950808437"
Jul 01 16:59:43.660523 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:43.660454    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-54cb84584-44p68" podStartSLOduration=33.817941698 podStartE2EDuration="38.660429047s" totalImagesPullingTime="4.842487349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:43.65795695 +0000 UTC m=+74.000297612" watchObservedRunningTime="2026-07-01 16:59:43.660429047 +0000 UTC m=+74.002769741"
Jul 01 16:59:44.429725 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:44.429687    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bl5g" event={"ID":"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9","Type":"ContainerStarted","Data":"3a500b35e03ed55ac95a902307b860fd29d7f6323c9a90e16712d9394f5e846b"}
Jul 01 16:59:51.458940 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.458904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bl5g" event={"ID":"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9","Type":"ContainerStarted","Data":"a0a77700d4e6629b5991d4e658c5bee1427a22fcb526958e107620dee3c2d17b"}
Jul 01 16:59:51.461086 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.461039    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9" event={"ID":"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a","Type":"ContainerStarted","Data":"6bc2be28f29fe82d8e9a10b15774a41925eeb59ebe987c294fb30c35eabc7852"}
Jul 01 16:59:51.461232 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.461103    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9" event={"ID":"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a","Type":"ContainerStarted","Data":"a6729077373b5e07b439d76bc306869f6a01033d5e988edac6e68390f73caf74"}
Jul 01 16:59:51.461232 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.461124    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9" event={"ID":"cdc7dfdb-ceb0-4a61-8e80-b2dcee9d0b2a","Type":"ContainerStarted","Data":"c460e8a232d7a9fc70e682f6687d1e3adc0ead543b48b49071f0633a685ee79c"}
Jul 01 16:59:51.462492 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.462467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9" event={"ID":"0fbeebf3-0e81-432b-8620-34613fc5a5de","Type":"ContainerStarted","Data":"4b688cb4657aa1bffc88186d56f05e184d53f9b5f1172eb3410c0054a247fdae"}
Jul 01 16:59:51.462698 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.462684    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:51.464568 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.464550    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9"
Jul 01 16:59:51.499739 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:51.499688    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d85f679d7-pz7s9" podStartSLOduration=3.801783232 podStartE2EDuration="11.499671866s" totalImagesPullingTime="7.697888634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:51.498604631 +0000 UTC m=+81.840945291" watchObservedRunningTime="2026-07-01 16:59:51.499671866 +0000 UTC m=+81.842012503"
Jul 01 16:59:52.469475 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:52.469440    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bl5g" event={"ID":"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9","Type":"ContainerStarted","Data":"90a61d3932ba22b4e45621bf0c412d6b9b48a5eeb2000f8e6ba71fc83bd49158"}
Jul 01 16:59:53.432884 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:53.432854    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vx286"
Jul 01 16:59:53.474182 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:53.474116    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d789c44f-2b9g9" podStartSLOduration=6.101806532 podStartE2EDuration="13.474093188s" totalImagesPullingTime="7.372286656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 16:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:51.528977132 +0000 UTC m=+81.871317793" watchObservedRunningTime="2026-07-01 16:59:53.474093188 +0000 UTC m=+83.816433851"
Jul 01 16:59:54.337704 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.337672    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5qkpl"]
Jul 01 16:59:54.344251 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.344223    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.352321 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.352298    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 16:59:54.352507 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.352490    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 16:59:54.353424 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.353391    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ttjh4\""
Jul 01 16:59:54.353508 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.353495    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 16:59:54.353771 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.353737    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 16:59:54.469967 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.469934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzpv\" (UniqueName: \"kubernetes.io/projected/dd6f2e1e-9296-4ed9-9250-888376d6f734-kube-api-access-sqzpv\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.469967 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.469976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.470207 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.469999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-accelerators-collector-config\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.470207 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.470035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f2e1e-9296-4ed9-9250-888376d6f734-metrics-client-ca\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.470207 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.470108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-wtmp\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.470207 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.470124    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-sys\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.470207 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.470137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-root\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.470207 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.470163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-textfile\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.470207 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.470195    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-tls\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.478850 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.478821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bl5g" event={"ID":"c9712d61-9858-4ec0-ae38-e3e16e9fa2d9","Type":"ContainerStarted","Data":"95876bff8f813e870fff285d03328148d68fb4e0fb00b615d0915937c5aa88a0"}
Jul 01 16:59:54.513654 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.513609    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5bl5g" podStartSLOduration=33.041913861 podStartE2EDuration="43.513595443s" totalImagesPullingTime="10.471681582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 16:59:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:54.512993709 +0000 UTC m=+84.855334381" watchObservedRunningTime="2026-07-01 16:59:54.513595443 +0000 UTC m=+84.855936103"
Jul 01 16:59:54.571287 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-textfile\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571437 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-tls\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571437 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571330    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzpv\" (UniqueName: \"kubernetes.io/projected/dd6f2e1e-9296-4ed9-9250-888376d6f734-kube-api-access-sqzpv\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571437 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:54.571414    2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 01 16:59:54.571593 ip-10-0-141-111 kubenswrapper[2556]: E0701 16:59:54.571470    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-tls podName:dd6f2e1e-9296-4ed9-9250-888376d6f734 nodeName:}" failed. No retries permitted until 2026-07-01 16:59:55.071453564 +0000 UTC m=+85.413794201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-tls") pod "node-exporter-5qkpl" (UID: "dd6f2e1e-9296-4ed9-9250-888376d6f734") : secret "node-exporter-tls" not found
Jul 01 16:59:54.571593 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571593 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571555    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-accelerators-collector-config\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571593 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571582    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-textfile\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571802 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571616    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f2e1e-9296-4ed9-9250-888376d6f734-metrics-client-ca\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571802 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-wtmp\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571802 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571697    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-sys\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571802 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571715    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-root\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.571802 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571795    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-sys\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.572048 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571806    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-wtmp\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.572048 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.571840    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/dd6f2e1e-9296-4ed9-9250-888376d6f734-root\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.572222 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.572203    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dd6f2e1e-9296-4ed9-9250-888376d6f734-metrics-client-ca\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.572277 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.572207    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-accelerators-collector-config\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.573798 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.573781    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:54.603167 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:54.603139    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzpv\" (UniqueName: \"kubernetes.io/projected/dd6f2e1e-9296-4ed9-9250-888376d6f734-kube-api-access-sqzpv\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:55.076562 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:55.076532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-tls\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:55.078817 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:55.078796    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/dd6f2e1e-9296-4ed9-9250-888376d6f734-node-exporter-tls\") pod \"node-exporter-5qkpl\" (UID: \"dd6f2e1e-9296-4ed9-9250-888376d6f734\") " pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:55.252753 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:55.252718    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5qkpl"
Jul 01 16:59:55.265072 ip-10-0-141-111 kubenswrapper[2556]: W0701 16:59:55.265023    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6f2e1e_9296_4ed9_9250_888376d6f734.slice/crio-f1f70b00190c3af92ac32f75c91a90945024fe72a862d80f21f5c56a64f64496 WatchSource:0}: Error finding container f1f70b00190c3af92ac32f75c91a90945024fe72a862d80f21f5c56a64f64496: Status 404 returned error can't find the container with id f1f70b00190c3af92ac32f75c91a90945024fe72a862d80f21f5c56a64f64496
Jul 01 16:59:55.483272 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:55.483241    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qkpl" event={"ID":"dd6f2e1e-9296-4ed9-9250-888376d6f734","Type":"ContainerStarted","Data":"f1f70b00190c3af92ac32f75c91a90945024fe72a862d80f21f5c56a64f64496"}
Jul 01 16:59:56.489101 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:56.489069    2556 generic.go:356] "Generic (PLEG): container finished" podID="dd6f2e1e-9296-4ed9-9250-888376d6f734" containerID="ff3c62218dbbd81e6e75fcd2ef930d7d63f6bc6e49f39c76cee60309b81142ce" exitCode=0
Jul 01 16:59:56.489506 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:56.489162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qkpl" event={"ID":"dd6f2e1e-9296-4ed9-9250-888376d6f734","Type":"ContainerDied","Data":"ff3c62218dbbd81e6e75fcd2ef930d7d63f6bc6e49f39c76cee60309b81142ce"}
Jul 01 16:59:57.494595 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:57.494564    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qkpl" event={"ID":"dd6f2e1e-9296-4ed9-9250-888376d6f734","Type":"ContainerStarted","Data":"d93c85bc1fb4933d34f3811e4893724deb267d1f7be60f42a877bfe5e321be6b"}
Jul 01 16:59:57.495007 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:57.494617    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qkpl" event={"ID":"dd6f2e1e-9296-4ed9-9250-888376d6f734","Type":"ContainerStarted","Data":"ee527d8d63f6e562128e2d676fe8b8d0e6e66c3a3020768c650e6c27aabceae6"}
Jul 01 16:59:57.536842 ip-10-0-141-111 kubenswrapper[2556]: I0701 16:59:57.536789    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5qkpl" podStartSLOduration=2.629737682 podStartE2EDuration="3.536774483s" totalImagesPullingTime="841.694474ms" totalInitContainerRuntime="65.342327ms" isStatefulPod=true podCreationTimestamp="2026-07-01 16:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 16:59:57.535360249 +0000 UTC m=+87.877700910" watchObservedRunningTime="2026-07-01 16:59:57.536774483 +0000 UTC m=+87.879115144"
Jul 01 17:00:14.432848 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:00:14.432819    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rkmmg"
Jul 01 17:00:27.848945 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:00:27.848915    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-44p68_fb1e46ce-6d0c-4bcd-b895-01f74c6a829f/cluster-monitoring-operator/0.log"
Jul 01 17:00:29.048310 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:00:29.048284    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qkpl_dd6f2e1e-9296-4ed9-9250-888376d6f734/init-textfile/0.log"
Jul 01 17:00:29.249825 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:00:29.249794    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qkpl_dd6f2e1e-9296-4ed9-9250-888376d6f734/node-exporter/0.log"
Jul 01 17:00:29.452709 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:00:29.452624    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qkpl_dd6f2e1e-9296-4ed9-9250-888376d6f734/kube-rbac-proxy/0.log"
Jul 01 17:03:30.006542 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:03:30.006511    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:03:30.010162 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:03:30.010133    2556 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 17:03:30.012140 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:03:30.012123    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:05:30.297243 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.297206    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-96gp7"]
Jul 01 17:05:30.299274 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.299255    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.302041 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.302018    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 17:05:30.302175 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.302018    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 17:05:30.303174 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.303154    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 01 17:05:30.303174 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.303172    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-vgwll\""
Jul 01 17:05:30.310586 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.310567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-96gp7"]
Jul 01 17:05:30.323961 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.323940    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-86fpm"]
Jul 01 17:05:30.326109 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.326094    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.328572 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.328549    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 01 17:05:30.328677 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.328623    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8sxwj\""
Jul 01 17:05:30.337840 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.337821    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-86fpm"]
Jul 01 17:05:30.422850 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.422821    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlw2l\" (UniqueName: \"kubernetes.io/projected/750af89d-bda4-4168-b871-194dc163c6d3-kube-api-access-qlw2l\") pod \"seaweedfs-64568bcd49-86fpm\" (UID: \"750af89d-bda4-4168-b871-194dc163c6d3\") " pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.423034 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.422869    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zd8\" (UniqueName: \"kubernetes.io/projected/a3a925f2-ba56-47ba-9801-b0aeab3191c8-kube-api-access-m9zd8\") pod \"llmisvc-controller-manager-67d95fdc48-96gp7\" (UID: \"a3a925f2-ba56-47ba-9801-b0aeab3191c8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.423034 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.422966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3a925f2-ba56-47ba-9801-b0aeab3191c8-cert\") pod \"llmisvc-controller-manager-67d95fdc48-96gp7\" (UID: \"a3a925f2-ba56-47ba-9801-b0aeab3191c8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.423034 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.423002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/750af89d-bda4-4168-b871-194dc163c6d3-data\") pod \"seaweedfs-64568bcd49-86fpm\" (UID: \"750af89d-bda4-4168-b871-194dc163c6d3\") " pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.523326 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.523276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlw2l\" (UniqueName: \"kubernetes.io/projected/750af89d-bda4-4168-b871-194dc163c6d3-kube-api-access-qlw2l\") pod \"seaweedfs-64568bcd49-86fpm\" (UID: \"750af89d-bda4-4168-b871-194dc163c6d3\") " pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.523502 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.523342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zd8\" (UniqueName: \"kubernetes.io/projected/a3a925f2-ba56-47ba-9801-b0aeab3191c8-kube-api-access-m9zd8\") pod \"llmisvc-controller-manager-67d95fdc48-96gp7\" (UID: \"a3a925f2-ba56-47ba-9801-b0aeab3191c8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.523502 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.523400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3a925f2-ba56-47ba-9801-b0aeab3191c8-cert\") pod \"llmisvc-controller-manager-67d95fdc48-96gp7\" (UID: \"a3a925f2-ba56-47ba-9801-b0aeab3191c8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.523502 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.523435    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/750af89d-bda4-4168-b871-194dc163c6d3-data\") pod \"seaweedfs-64568bcd49-86fpm\" (UID: \"750af89d-bda4-4168-b871-194dc163c6d3\") " pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.523830 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.523793    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/750af89d-bda4-4168-b871-194dc163c6d3-data\") pod \"seaweedfs-64568bcd49-86fpm\" (UID: \"750af89d-bda4-4168-b871-194dc163c6d3\") " pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.525744 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.525725    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3a925f2-ba56-47ba-9801-b0aeab3191c8-cert\") pod \"llmisvc-controller-manager-67d95fdc48-96gp7\" (UID: \"a3a925f2-ba56-47ba-9801-b0aeab3191c8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.533322 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.533297    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlw2l\" (UniqueName: \"kubernetes.io/projected/750af89d-bda4-4168-b871-194dc163c6d3-kube-api-access-qlw2l\") pod \"seaweedfs-64568bcd49-86fpm\" (UID: \"750af89d-bda4-4168-b871-194dc163c6d3\") " pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.533932 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.533914    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zd8\" (UniqueName: \"kubernetes.io/projected/a3a925f2-ba56-47ba-9801-b0aeab3191c8-kube-api-access-m9zd8\") pod \"llmisvc-controller-manager-67d95fdc48-96gp7\" (UID: \"a3a925f2-ba56-47ba-9801-b0aeab3191c8\") " pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.609956 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.609862    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:30.635206 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.635169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:30.751761 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.751727    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67d95fdc48-96gp7"]
Jul 01 17:05:30.755162 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:05:30.755138    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda3a925f2_ba56_47ba_9801_b0aeab3191c8.slice/crio-afeba6c9541332f652482fc3e67e743393d540b9c223817eab0e9c565c58f9ef WatchSource:0}: Error finding container afeba6c9541332f652482fc3e67e743393d540b9c223817eab0e9c565c58f9ef: Status 404 returned error can't find the container with id afeba6c9541332f652482fc3e67e743393d540b9c223817eab0e9c565c58f9ef
Jul 01 17:05:30.756469 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.756451    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:05:30.779828 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:30.779804    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-86fpm"]
Jul 01 17:05:30.782576 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:05:30.782554    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750af89d_bda4_4168_b871_194dc163c6d3.slice/crio-c73cfb6791ab08955bcced880fe97c66fe5d0ceac52fb0f1f1e0510b99e1e1bf WatchSource:0}: Error finding container c73cfb6791ab08955bcced880fe97c66fe5d0ceac52fb0f1f1e0510b99e1e1bf: Status 404 returned error can't find the container with id c73cfb6791ab08955bcced880fe97c66fe5d0ceac52fb0f1f1e0510b99e1e1bf
Jul 01 17:05:31.570720 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:31.570669    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-86fpm" event={"ID":"750af89d-bda4-4168-b871-194dc163c6d3","Type":"ContainerStarted","Data":"c73cfb6791ab08955bcced880fe97c66fe5d0ceac52fb0f1f1e0510b99e1e1bf"}
Jul 01 17:05:31.571970 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:31.571940    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7" event={"ID":"a3a925f2-ba56-47ba-9801-b0aeab3191c8","Type":"ContainerStarted","Data":"afeba6c9541332f652482fc3e67e743393d540b9c223817eab0e9c565c58f9ef"}
Jul 01 17:05:33.580344 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:33.580308    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7" event={"ID":"a3a925f2-ba56-47ba-9801-b0aeab3191c8","Type":"ContainerStarted","Data":"daf0a5a47a1f65ef336027867d48334e8c9d715ad6070b05ba8f3f8a9a02367a"}
Jul 01 17:05:33.580757 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:33.580415    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:05:33.625911 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:33.625857    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7" podStartSLOduration=1.6826514609999998 podStartE2EDuration="3.625838962s" totalImagesPullingTime="1.943187501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:05:33.625041371 +0000 UTC m=+423.967382032" watchObservedRunningTime="2026-07-01 17:05:33.625838962 +0000 UTC m=+423.968179648"
Jul 01 17:05:35.589450 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:35.589417    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-86fpm" event={"ID":"750af89d-bda4-4168-b871-194dc163c6d3","Type":"ContainerStarted","Data":"21ded7c1379cd130285f65fd5ee2733eab1e9911423733250bebdd3833cf80c0"}
Jul 01 17:05:35.589841 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:35.589553    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:05:35.612582 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:35.612534    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-86fpm" podStartSLOduration=1.588663282 podStartE2EDuration="5.612519326s" totalImagesPullingTime="4.023856044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:05:35.610640247 +0000 UTC m=+425.952980908" watchObservedRunningTime="2026-07-01 17:05:35.612519326 +0000 UTC m=+425.954859985"
Jul 01 17:05:41.594877 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:05:41.594848    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-86fpm"
Jul 01 17:06:04.587358 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:04.587327    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67d95fdc48-96gp7"
Jul 01 17:06:40.386914 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.386878    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xrpwl"]
Jul 01 17:06:40.389075 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.389041    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:40.396771 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.396748    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jpgg8\""
Jul 01 17:06:40.396921 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.396904    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 17:06:40.405482 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.405460    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xrpwl"]
Jul 01 17:06:40.417190 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.417166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-cert\") pod \"odh-model-controller-84b54b7f97-xrpwl\" (UID: \"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5\") " pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:40.417305 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.417232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hllgb\" (UniqueName: \"kubernetes.io/projected/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-kube-api-access-hllgb\") pod \"odh-model-controller-84b54b7f97-xrpwl\" (UID: \"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5\") " pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:40.518407 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.518351    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-cert\") pod \"odh-model-controller-84b54b7f97-xrpwl\" (UID: \"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5\") " pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:40.518586 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:06:40.518483    2556 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 01 17:06:40.518586 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.518483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hllgb\" (UniqueName: \"kubernetes.io/projected/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-kube-api-access-hllgb\") pod \"odh-model-controller-84b54b7f97-xrpwl\" (UID: \"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5\") " pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:40.518586 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:06:40.518542    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-cert podName:8ff7439b-31b9-452d-9f44-1e0a3bea8ca5 nodeName:}" failed. No retries permitted until 2026-07-01 17:06:41.018526042 +0000 UTC m=+491.360866679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-cert") pod "odh-model-controller-84b54b7f97-xrpwl" (UID: "8ff7439b-31b9-452d-9f44-1e0a3bea8ca5") : secret "odh-model-controller-webhook-cert" not found
Jul 01 17:06:40.529501 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:40.529475    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hllgb\" (UniqueName: \"kubernetes.io/projected/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-kube-api-access-hllgb\") pod \"odh-model-controller-84b54b7f97-xrpwl\" (UID: \"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5\") " pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:41.021685 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:41.021644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-cert\") pod \"odh-model-controller-84b54b7f97-xrpwl\" (UID: \"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5\") " pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:41.023961 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:41.023938    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ff7439b-31b9-452d-9f44-1e0a3bea8ca5-cert\") pod \"odh-model-controller-84b54b7f97-xrpwl\" (UID: \"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5\") " pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:41.298662 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:41.298568    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:41.426996 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:41.426967    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-xrpwl"]
Jul 01 17:06:41.429623 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:06:41.429597    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff7439b_31b9_452d_9f44_1e0a3bea8ca5.slice/crio-d62a915cfa6c9200819c1143fdafc5092e16953af8ef2c7891ffe85360eebf35 WatchSource:0}: Error finding container d62a915cfa6c9200819c1143fdafc5092e16953af8ef2c7891ffe85360eebf35: Status 404 returned error can't find the container with id d62a915cfa6c9200819c1143fdafc5092e16953af8ef2c7891ffe85360eebf35
Jul 01 17:06:41.822379 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:41.822341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xrpwl" event={"ID":"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5","Type":"ContainerStarted","Data":"d62a915cfa6c9200819c1143fdafc5092e16953af8ef2c7891ffe85360eebf35"}
Jul 01 17:06:44.836133 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:44.836093    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-xrpwl" event={"ID":"8ff7439b-31b9-452d-9f44-1e0a3bea8ca5","Type":"ContainerStarted","Data":"b54a48a7abe9b7b29f65d0967bc166aba425381bdf9baf992319553ea11145ee"}
Jul 01 17:06:44.836548 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:44.836215    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:06:44.856758 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:44.856663    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-xrpwl" podStartSLOduration=1.694515609 podStartE2EDuration="4.8566477s" totalImagesPullingTime="3.162132091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:06:44.856298141 +0000 UTC m=+495.198638805" watchObservedRunningTime="2026-07-01 17:06:44.8566477 +0000 UTC m=+495.198988392"
Jul 01 17:06:55.842878 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:06:55.842848    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-xrpwl"
Jul 01 17:08:27.598998 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.598965    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"]
Jul 01 17:08:27.601258 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.601239    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.605363 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.605342    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:08:27.605538 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.605520    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g2mrd\""
Jul 01 17:08:27.605670 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.605656    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:08:27.606255 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.606237    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:08:27.606560 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.606541    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:08:27.607514 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.607494    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9a882-predictor-serving-cert\""
Jul 01 17:08:27.611206 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.611188    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-9a882-kube-rbac-proxy-sar-config\""
Jul 01 17:08:27.623690 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.623670    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"]
Jul 01 17:08:27.694974 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.694943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.694974 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.694974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.695204 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.694998    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.695204 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.695103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrbh\" (UniqueName: \"kubernetes.io/projected/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-api-access-pqrbh\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.695204 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.695133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.796421 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.796384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrbh\" (UniqueName: \"kubernetes.io/projected/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-api-access-pqrbh\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.796421 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.796429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.796709 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.796461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.796709 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.796480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.796709 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.796512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.796889 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.796874    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.797360 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.797335    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.797422 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.797372    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.798828 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.798810    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-proxy-tls\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.804983 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.804960    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrbh\" (UniqueName: \"kubernetes.io/projected/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-api-access-pqrbh\") pod \"isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:27.911910 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:27.911816    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:28.045968 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:28.045943    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"]
Jul 01 17:08:28.048121 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:08:28.048095    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2b7f1f_a903_4a32_b77c_a2e894fec0d7.slice/crio-5b0ec0eb070fce2030e00f242597e32745e3b2177c2d5d16322ad951e95c0861 WatchSource:0}: Error finding container 5b0ec0eb070fce2030e00f242597e32745e3b2177c2d5d16322ad951e95c0861: Status 404 returned error can't find the container with id 5b0ec0eb070fce2030e00f242597e32745e3b2177c2d5d16322ad951e95c0861
Jul 01 17:08:28.196364 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:28.196281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerStarted","Data":"5b0ec0eb070fce2030e00f242597e32745e3b2177c2d5d16322ad951e95c0861"}
Jul 01 17:08:30.591615 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:30.591583    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:08:30.592188 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:30.592170    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:08:32.062644 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:32.062621    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:08:32.215072 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:32.215027    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerStarted","Data":"5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e"}
Jul 01 17:08:36.228784 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:36.228750    2556 generic.go:356] "Generic (PLEG): container finished" podID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerID="5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e" exitCode=0
Jul 01 17:08:36.229174 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:36.228828    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerDied","Data":"5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e"}
Jul 01 17:08:50.304940 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:50.304900    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerStarted","Data":"765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8"}
Jul 01 17:08:52.315421 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:52.315390    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerStarted","Data":"5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2"}
Jul 01 17:08:56.335013 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:56.334979    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerStarted","Data":"0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52"}
Jul 01 17:08:56.335399 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:56.335242    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:56.335399 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:56.335367    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:56.336649 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:56.336610    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:08:57.338257 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:57.338226    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:57.338799 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:57.338345    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:08:57.339389 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:57.339352    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:08:57.342405 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:57.342386    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:08:57.374484 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:57.374410    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podStartSLOduration=3.330179611 podStartE2EDuration="30.374390175s" totalImagesPullingTime="23.314533884s" totalInitContainerRuntime="3.72967668s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:08:27 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:08:56.365016871 +0000 UTC m=+626.707357531" watchObservedRunningTime="2026-07-01 17:08:57.374390175 +0000 UTC m=+627.716730837"
Jul 01 17:08:58.341666 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:58.341629    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:08:58.342133 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:58.341917    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:08:59.348336 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:59.348297    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:08:59.348897 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:08:59.348873    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:09.348664 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:09.348611    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:09:09.349162 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:09.349109    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:19.349267 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:19.349218    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:09:19.351149 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:19.349639    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:29.348752 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:29.348697    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:09:29.349324 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:29.349217    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:39.349161 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:39.349102    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:09:39.349634 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:39.349611    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:49.349240 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:49.349187    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:09:49.349763 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:49.349709    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:09:59.348327 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:59.348285    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:09:59.348749 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:09:59.348681    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:09.349302 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:09.349268    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:10:09.349717 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:09.349400    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:10:22.769993 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.769916    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"]
Jul 01 17:10:22.770494 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.770199    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" containerID="cri-o://765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8" gracePeriod=30
Jul 01 17:10:22.770494 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.770251    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy" containerID="cri-o://5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2" gracePeriod=30
Jul 01 17:10:22.770494 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.770229    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" containerID="cri-o://0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52" gracePeriod=30
Jul 01 17:10:22.786012 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.785984    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"]
Jul 01 17:10:22.789860 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.789830    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:22.792577 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.792557    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-89809-predictor-serving-cert\""
Jul 01 17:10:22.792656 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.792614    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-89809-kube-rbac-proxy-sar-config\""
Jul 01 17:10:22.806409 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.806383    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"]
Jul 01 17:10:22.896144 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.896113    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"]
Jul 01 17:10:22.899887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.899864    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:22.902529 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.902509    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-89809-predictor-serving-cert\""
Jul 01 17:10:22.902657 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.902564    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-89809-kube-rbac-proxy-sar-config\""
Jul 01 17:10:22.906042 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.906019    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c9d5-258b-4777-a905-83df72e038f2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:22.906156 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.906071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:22.906205 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.906163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:22.906416 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.906220    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:22.906416 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.906299    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ps8\" (UniqueName: \"kubernetes.io/projected/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-api-access-v5ps8\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:22.911091 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:22.911053    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"]
Jul 01 17:10:23.007033 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c9d5-258b-4777-a905-83df72e038f2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.007286 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007042    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.007286 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.007286 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007131    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.007286 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.007286 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5a468c-c5ff-4faf-b323-2a213450d04f-proxy-tls\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.007286 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:10:23.007250    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-serving-cert: secret "isvc-sklearn-graph-raw-89809-predictor-serving-cert" not found
Jul 01 17:10:23.007636 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5a468c-c5ff-4faf-b323-2a213450d04f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.007636 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:10:23.007321    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls podName:f4c0c9d5-258b-4777-a905-83df72e038f2 nodeName:}" failed. No retries permitted until 2026-07-01 17:10:23.507303668 +0000 UTC m=+713.849644332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls") pod "isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" (UID: "f4c0c9d5-258b-4777-a905-83df72e038f2") : secret "isvc-sklearn-graph-raw-89809-predictor-serving-cert" not found
Jul 01 17:10:23.007636 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.007636 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007470    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c9d5-258b-4777-a905-83df72e038f2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.007636 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007510    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2nz\" (UniqueName: \"kubernetes.io/projected/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-api-access-6r2nz\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.007636 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ps8\" (UniqueName: \"kubernetes.io/projected/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-api-access-v5ps8\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.007918 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007811    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.007974 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.007957    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.017605 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.017581    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ps8\" (UniqueName: \"kubernetes.io/projected/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-api-access-v5ps8\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.108597 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.108501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2nz\" (UniqueName: \"kubernetes.io/projected/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-api-access-6r2nz\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.108597 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.108581    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.108817 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.108606    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.108817 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.108624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5a468c-c5ff-4faf-b323-2a213450d04f-proxy-tls\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.108817 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.108644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5a468c-c5ff-4faf-b323-2a213450d04f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.109172 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.109144    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5a468c-c5ff-4faf-b323-2a213450d04f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.109390 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.109370    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.109464 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.109370    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.111187 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.111170    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5a468c-c5ff-4faf-b323-2a213450d04f-proxy-tls\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.117722 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.117699    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2nz\" (UniqueName: \"kubernetes.io/projected/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-api-access-6r2nz\") pod \"isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.209831 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.209800    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:23.343256 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.343231    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"]
Jul 01 17:10:23.345521 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:10:23.345486    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd5a468c_c5ff_4faf_b323_2a213450d04f.slice/crio-f061fba169c1b3f377c6add1a6159078f62ea49cea225c86a5c92811a58b3973 WatchSource:0}: Error finding container f061fba169c1b3f377c6add1a6159078f62ea49cea225c86a5c92811a58b3973: Status 404 returned error can't find the container with id f061fba169c1b3f377c6add1a6159078f62ea49cea225c86a5c92811a58b3973
Jul 01 17:10:23.511604 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.511567    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.513932 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.513907    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls\") pod \"isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.647236 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.647142    2556 generic.go:356] "Generic (PLEG): container finished" podID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerID="5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2" exitCode=2
Jul 01 17:10:23.647236 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.647223    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerDied","Data":"5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2"}
Jul 01 17:10:23.648760 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.648735    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerStarted","Data":"22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9"}
Jul 01 17:10:23.648920 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.648788    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerStarted","Data":"f061fba169c1b3f377c6add1a6159078f62ea49cea225c86a5c92811a58b3973"}
Jul 01 17:10:23.699677 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.699637    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:23.841744 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:23.841713    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"]
Jul 01 17:10:23.844711 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:10:23.844679    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c0c9d5_258b_4777_a905_83df72e038f2.slice/crio-fb6fb29f1032206a625ad5d02290da696432e1d1ec37ea00cdb0352c819f891f WatchSource:0}: Error finding container fb6fb29f1032206a625ad5d02290da696432e1d1ec37ea00cdb0352c819f891f: Status 404 returned error can't find the container with id fb6fb29f1032206a625ad5d02290da696432e1d1ec37ea00cdb0352c819f891f
Jul 01 17:10:24.653735 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:24.653703    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerStarted","Data":"179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933"}
Jul 01 17:10:24.653920 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:24.653762    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerStarted","Data":"fb6fb29f1032206a625ad5d02290da696432e1d1ec37ea00cdb0352c819f891f"}
Jul 01 17:10:27.339426 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:27.339384    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 01 17:10:27.668630 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:27.668554    2556 generic.go:356] "Generic (PLEG): container finished" podID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerID="765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8" exitCode=0
Jul 01 17:10:27.668775 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:27.668627    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerDied","Data":"765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8"}
Jul 01 17:10:27.670100 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:27.670079    2556 generic.go:356] "Generic (PLEG): container finished" podID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerID="22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9" exitCode=0
Jul 01 17:10:27.670200 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:27.670107    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerDied","Data":"22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9"}
Jul 01 17:10:28.678132 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:28.678055    2556 generic.go:356] "Generic (PLEG): container finished" podID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerID="179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933" exitCode=0
Jul 01 17:10:28.678132 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:28.678099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerDied","Data":"179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933"}
Jul 01 17:10:29.348683 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:29.348633    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:10:29.349007 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:29.348969    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:30.694824 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:30.694774    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerStarted","Data":"22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8"}
Jul 01 17:10:30.695313 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:30.694886    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerStarted","Data":"66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683"}
Jul 01 17:10:30.695415 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:30.695389    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:30.717494 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:30.717443    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podStartSLOduration=4.868674065 podStartE2EDuration="8.717424297s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.848750232s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:10:30.716433411 +0000 UTC m=+721.058774074" watchObservedRunningTime="2026-07-01 17:10:30.717424297 +0000 UTC m=+721.059764961"
Jul 01 17:10:31.699596 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:31.699570    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:31.701154 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:31.701126    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:10:32.338827 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:32.338783    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 01 17:10:32.704251 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:32.704163    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:10:37.338795 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:37.338750    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 01 17:10:37.339302 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:37.338899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:10:37.708751 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:37.708678    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:10:37.709268 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:37.709243    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:10:39.349270 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:39.349217    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:10:39.349733 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:39.349528    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:42.339001 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:42.338939    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 01 17:10:47.339367 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:47.339316    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 01 17:10:47.709376 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:47.709286    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:10:48.771149 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:48.771116    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerStarted","Data":"500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f"}
Jul 01 17:10:48.771527 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:48.771178    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerStarted","Data":"bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d"}
Jul 01 17:10:48.771527 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:48.771368    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:48.793666 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:48.793614    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podStartSLOduration=2.42833037 podStartE2EDuration="26.793578668s" totalImagesPullingTime="20.49515577s" totalInitContainerRuntime="3.870092528s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:10:48.791984727 +0000 UTC m=+739.134325409" watchObservedRunningTime="2026-07-01 17:10:48.793578668 +0000 UTC m=+739.135919328"
Jul 01 17:10:49.349154 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:49.349103    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 01 17:10:49.349357 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:49.349265    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:10:49.349572 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:49.349545    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:10:49.349656 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:49.349642    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:10:49.775419 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:49.775388    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:49.776789 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:49.776759    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:10:50.778436 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:50.778405    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:10:52.338642 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.338600    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Jul 01 17:10:52.921077 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.921036    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:10:52.955943 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.955913    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kserve-provision-location\") pod \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") "
Jul 01 17:10:52.956101 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.955999    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-rbac-proxy-sar-config\") pod \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") "
Jul 01 17:10:52.956101 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.956035    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-cabundle-cert\") pod \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") "
Jul 01 17:10:52.956101 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.956084    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqrbh\" (UniqueName: \"kubernetes.io/projected/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-api-access-pqrbh\") pod \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") "
Jul 01 17:10:52.956223 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.956115    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-proxy-tls\") pod \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\" (UID: \"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7\") "
Jul 01 17:10:52.956269 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.956245    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" (UID: "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:10:52.956373 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.956355    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:10:52.956421 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.956373    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" (UID: "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:10:52.956421 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.956381    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" (UID: "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:10:52.958358 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.958328    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-api-access-pqrbh" (OuterVolumeSpecName: "kube-api-access-pqrbh") pod "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" (UID: "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7"). InnerVolumeSpecName "kube-api-access-pqrbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:10:52.958447 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:52.958343    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" (UID: "2f2b7f1f-a903-4a32-b77c-a2e894fec0d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:10:53.056987 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.056898    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:10:53.056987 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.056927    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:10:53.056987 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.056939    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrbh\" (UniqueName: \"kubernetes.io/projected/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-kube-api-access-pqrbh\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:10:53.056987 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.056950    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:10:53.791734 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.791703    2556 generic.go:356] "Generic (PLEG): container finished" podID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerID="0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52" exitCode=0
Jul 01 17:10:53.792211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.791754    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerDied","Data":"0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52"}
Jul 01 17:10:53.792211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.791764    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"
Jul 01 17:10:53.792211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.791808    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp" event={"ID":"2f2b7f1f-a903-4a32-b77c-a2e894fec0d7","Type":"ContainerDied","Data":"5b0ec0eb070fce2030e00f242597e32745e3b2177c2d5d16322ad951e95c0861"}
Jul 01 17:10:53.792211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.791840    2556 scope.go:117] "RemoveContainer" containerID="0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52"
Jul 01 17:10:53.800387 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.800369    2556 scope.go:117] "RemoveContainer" containerID="5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2"
Jul 01 17:10:53.807180 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.807164    2556 scope.go:117] "RemoveContainer" containerID="765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8"
Jul 01 17:10:53.813844 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.813824    2556 scope.go:117] "RemoveContainer" containerID="5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e"
Jul 01 17:10:53.820052 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.820032    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"]
Jul 01 17:10:53.821261 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.821244    2556 scope.go:117] "RemoveContainer" containerID="0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52"
Jul 01 17:10:53.821548 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:10:53.821523    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52\": container with ID starting with 0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52 not found: ID does not exist" containerID="0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52"
Jul 01 17:10:53.821638 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.821563    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52"} err="failed to get container status \"0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52\": rpc error: code = NotFound desc = could not find container \"0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52\": container with ID starting with 0648b58fe0b76b5301214fa3079f7e63d19536279fa57d9522d17e682caafc52 not found: ID does not exist"
Jul 01 17:10:53.821638 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.821605    2556 scope.go:117] "RemoveContainer" containerID="5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2"
Jul 01 17:10:53.821869 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:10:53.821845    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2\": container with ID starting with 5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2 not found: ID does not exist" containerID="5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2"
Jul 01 17:10:53.821914 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.821871    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2"} err="failed to get container status \"5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2\": rpc error: code = NotFound desc = could not find container \"5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2\": container with ID starting with 5fe4b3352650c7d1b076206246c56ce9de84457922a30ebad47883e1368e73b2 not found: ID does not exist"
Jul 01 17:10:53.821914 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.821885    2556 scope.go:117] "RemoveContainer" containerID="765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8"
Jul 01 17:10:53.822135 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:10:53.822108    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8\": container with ID starting with 765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8 not found: ID does not exist" containerID="765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8"
Jul 01 17:10:53.822188 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.822140    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8"} err="failed to get container status \"765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8\": rpc error: code = NotFound desc = could not find container \"765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8\": container with ID starting with 765e29930d1573d07197958511114f4d6df7befce3389cd1b77a8f0a6a4f7ee8 not found: ID does not exist"
Jul 01 17:10:53.822188 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.822154    2556 scope.go:117] "RemoveContainer" containerID="5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e"
Jul 01 17:10:53.822410 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:10:53.822393    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e\": container with ID starting with 5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e not found: ID does not exist" containerID="5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e"
Jul 01 17:10:53.822478 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.822416    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e"} err="failed to get container status \"5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e\": rpc error: code = NotFound desc = could not find container \"5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e\": container with ID starting with 5657e504c679dfc3c85fa689df52e4f360c4be69393410f76d1a727edd9b7e2e not found: ID does not exist"
Jul 01 17:10:53.825037 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:53.825018    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-9a882-predictor-cd7c84dbb-l7nsp"]
Jul 01 17:10:54.073352 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:54.073276    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" path="/var/lib/kubelet/pods/2f2b7f1f-a903-4a32-b77c-a2e894fec0d7/volumes"
Jul 01 17:10:55.782677 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:55.782649    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:10:55.783164 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:55.783139    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:10:57.709852 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:10:57.709813    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:11:05.783129 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:05.783086    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:11:07.709922 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:07.709885    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:11:15.783208 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:15.783170    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:11:17.710171 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:17.710132    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:11:25.783281 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:25.783241    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:11:27.710131 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:27.710094    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:11:35.783754 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:35.783714    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:11:37.709890 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:37.709853    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:11:43.069157 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:43.069111    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 01 17:11:45.783149 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:45.783105    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:11:53.069733 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:53.069704    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:11:55.784266 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:11:55.784231    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:12:23.086126 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.086097    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"]
Jul 01 17:12:23.086595 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.086372    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container" containerID="cri-o://66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683" gracePeriod=30
Jul 01 17:12:23.086595 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.086376    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kube-rbac-proxy" containerID="cri-o://22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8" gracePeriod=30
Jul 01 17:12:23.107216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107191    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"]
Jul 01 17:12:23.107467 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107456    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent"
Jul 01 17:12:23.107516 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107469    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent"
Jul 01 17:12:23.107516 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107481    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container"
Jul 01 17:12:23.107516 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107487    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container"
Jul 01 17:12:23.107516 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107493    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy"
Jul 01 17:12:23.107516 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107498    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy"
Jul 01 17:12:23.107516 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107513    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="storage-initializer"
Jul 01 17:12:23.107516 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107518    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="storage-initializer"
Jul 01 17:12:23.107717 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107563    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kube-rbac-proxy"
Jul 01 17:12:23.107717 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107571    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="agent"
Jul 01 17:12:23.107717 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.107578    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2b7f1f-a903-4a32-b77c-a2e894fec0d7" containerName="kserve-container"
Jul 01 17:12:23.109639 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.109619    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.113037 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.112851    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e08af-predictor-serving-cert\""
Jul 01 17:12:23.113037 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.112887    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e08af-kube-rbac-proxy-sar-config\""
Jul 01 17:12:23.126833 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.126803    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"]
Jul 01 17:12:23.138534 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.138505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.138644 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.138557    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.138644 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.138607    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tnl5\" (UniqueName: \"kubernetes.io/projected/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-api-access-8tnl5\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.138827 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.138807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a40b5d-a681-4b80-942b-dd06dc63769f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.138961 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.138936    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.195273 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.195245    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"]
Jul 01 17:12:23.197512 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.197495    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.200269 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.200248    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e08af-predictor-serving-cert\""
Jul 01 17:12:23.200418 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.200246    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-e08af-kube-rbac-proxy-sar-config\""
Jul 01 17:12:23.210524 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.210504    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"]
Jul 01 17:12:23.240195 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qjd\" (UniqueName: \"kubernetes.io/projected/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-api-access-d5qjd\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.240401 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240219    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.240401 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240253    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.240401 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240280    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.240401 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.240401 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:23.240342    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-e08af-predictor-serving-cert" not found
Jul 01 17:12:23.241323 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:23.240410    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls podName:c5a40b5d-a681-4b80-942b-dd06dc63769f nodeName:}" failed. No retries permitted until 2026-07-01 17:12:23.740390103 +0000 UTC m=+834.082730742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" (UID: "c5a40b5d-a681-4b80-942b-dd06dc63769f") : secret "isvc-sklearn-graph-raw-hpa-e08af-predictor-serving-cert" not found
Jul 01 17:12:23.241323 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.241323 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240640    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tnl5\" (UniqueName: \"kubernetes.io/projected/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-api-access-8tnl5\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.241323 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240679    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.241323 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a40b5d-a681-4b80-942b-dd06dc63769f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.241323 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.240797    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.241323 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.241104    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.241666 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.241397    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a40b5d-a681-4b80-942b-dd06dc63769f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.241666 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.241523    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.253710 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.253681    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tnl5\" (UniqueName: \"kubernetes.io/projected/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-api-access-8tnl5\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.262519 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.262483    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"]
Jul 01 17:12:23.262774 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.262749    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" containerID="cri-o://bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d" gracePeriod=30
Jul 01 17:12:23.262866 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.262799    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kube-rbac-proxy" containerID="cri-o://500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f" gracePeriod=30
Jul 01 17:12:23.341411 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.341337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qjd\" (UniqueName: \"kubernetes.io/projected/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-api-access-d5qjd\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.341411 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.341400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.341623 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.341432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.341623 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.341458    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.341623 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.341488    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.341623 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:23.341607    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-e08af-predictor-serving-cert" not found
Jul 01 17:12:23.341773 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:23.341676    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls podName:ec6d77c8-9be9-404e-8a0f-00503f939b4d nodeName:}" failed. No retries permitted until 2026-07-01 17:12:23.841650483 +0000 UTC m=+834.183991341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" (UID: "ec6d77c8-9be9-404e-8a0f-00503f939b4d") : secret "isvc-xgboost-graph-raw-hpa-e08af-predictor-serving-cert" not found
Jul 01 17:12:23.341910 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.341889    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.342203 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.342186    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.342253 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.342185    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.351947 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.351922    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qjd\" (UniqueName: \"kubernetes.io/projected/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-api-access-d5qjd\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.744754 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.744725    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.747225 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.747189    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:23.845110 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.845048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:23.847397 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:23.847374    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:24.023095 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.022995    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:24.107355 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.107326    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:24.126596 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.126562    2556 generic.go:356] "Generic (PLEG): container finished" podID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerID="22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8" exitCode=2
Jul 01 17:12:24.126746 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.126667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerDied","Data":"22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8"}
Jul 01 17:12:24.128767 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.128738    2556 generic.go:356] "Generic (PLEG): container finished" podID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerID="500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f" exitCode=2
Jul 01 17:12:24.128892 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.128808    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerDied","Data":"500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f"}
Jul 01 17:12:24.157248 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.156830    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"]
Jul 01 17:12:24.159523 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:12:24.159406    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a40b5d_a681_4b80_942b_dd06dc63769f.slice/crio-e3e3e0668c0a19e383be357a0fbd38c2c4db5e4e5747d6da7dd5633f37ab30df WatchSource:0}: Error finding container e3e3e0668c0a19e383be357a0fbd38c2c4db5e4e5747d6da7dd5633f37ab30df: Status 404 returned error can't find the container with id e3e3e0668c0a19e383be357a0fbd38c2c4db5e4e5747d6da7dd5633f37ab30df
Jul 01 17:12:24.254622 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:24.254588    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"]
Jul 01 17:12:24.257454 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:12:24.257426    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6d77c8_9be9_404e_8a0f_00503f939b4d.slice/crio-44124234e835b83789985b19f9cdeacb7423872ebe191a00e29b5373ed3bdad6 WatchSource:0}: Error finding container 44124234e835b83789985b19f9cdeacb7423872ebe191a00e29b5373ed3bdad6: Status 404 returned error can't find the container with id 44124234e835b83789985b19f9cdeacb7423872ebe191a00e29b5373ed3bdad6
Jul 01 17:12:25.134205 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:25.134164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerStarted","Data":"0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a"}
Jul 01 17:12:25.134670 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:25.134260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerStarted","Data":"e3e3e0668c0a19e383be357a0fbd38c2c4db5e4e5747d6da7dd5633f37ab30df"}
Jul 01 17:12:25.135584 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:25.135563    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerStarted","Data":"505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e"}
Jul 01 17:12:25.135675 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:25.135618    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerStarted","Data":"44124234e835b83789985b19f9cdeacb7423872ebe191a00e29b5373ed3bdad6"}
Jul 01 17:12:25.779333 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:25.779289    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 01 17:12:25.783553 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:25.783528    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 01 17:12:26.907609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.907586    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:12:26.973490 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.973459    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r2nz\" (UniqueName: \"kubernetes.io/projected/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-api-access-6r2nz\") pod \"dd5a468c-c5ff-4faf-b323-2a213450d04f\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") "
Jul 01 17:12:26.973654 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.973578    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-rbac-proxy-sar-config\") pod \"dd5a468c-c5ff-4faf-b323-2a213450d04f\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") "
Jul 01 17:12:26.973654 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.973614    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-cabundle-cert\") pod \"dd5a468c-c5ff-4faf-b323-2a213450d04f\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") "
Jul 01 17:12:26.973654 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.973643    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5a468c-c5ff-4faf-b323-2a213450d04f-proxy-tls\") pod \"dd5a468c-c5ff-4faf-b323-2a213450d04f\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") "
Jul 01 17:12:26.973801 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.973673    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5a468c-c5ff-4faf-b323-2a213450d04f-kserve-provision-location\") pod \"dd5a468c-c5ff-4faf-b323-2a213450d04f\" (UID: \"dd5a468c-c5ff-4faf-b323-2a213450d04f\") "
Jul 01 17:12:26.973973 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.973940    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd5a468c-c5ff-4faf-b323-2a213450d04f" (UID: "dd5a468c-c5ff-4faf-b323-2a213450d04f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:12:26.974115 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.973988    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd5a468c-c5ff-4faf-b323-2a213450d04f" (UID: "dd5a468c-c5ff-4faf-b323-2a213450d04f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:12:26.974115 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.974038    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd5a468c-c5ff-4faf-b323-2a213450d04f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd5a468c-c5ff-4faf-b323-2a213450d04f" (UID: "dd5a468c-c5ff-4faf-b323-2a213450d04f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:12:26.975678 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.975644    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd5a468c-c5ff-4faf-b323-2a213450d04f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd5a468c-c5ff-4faf-b323-2a213450d04f" (UID: "dd5a468c-c5ff-4faf-b323-2a213450d04f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:12:26.975757 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:26.975684    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-api-access-6r2nz" (OuterVolumeSpecName: "kube-api-access-6r2nz") pod "dd5a468c-c5ff-4faf-b323-2a213450d04f" (UID: "dd5a468c-c5ff-4faf-b323-2a213450d04f"). InnerVolumeSpecName "kube-api-access-6r2nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:12:27.074898 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.074811    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:27.074898 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.074840    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd5a468c-c5ff-4faf-b323-2a213450d04f-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:27.074898 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.074850    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd5a468c-c5ff-4faf-b323-2a213450d04f-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:27.074898 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.074858    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd5a468c-c5ff-4faf-b323-2a213450d04f-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:27.074898 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.074866    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r2nz\" (UniqueName: \"kubernetes.io/projected/dd5a468c-c5ff-4faf-b323-2a213450d04f-kube-api-access-6r2nz\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:27.144376 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.144339    2556 generic.go:356] "Generic (PLEG): container finished" podID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerID="bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d" exitCode=0
Jul 01 17:12:27.144511 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.144400    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"
Jul 01 17:12:27.144511 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.144428    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerDied","Data":"bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d"}
Jul 01 17:12:27.144511 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.144493    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv" event={"ID":"dd5a468c-c5ff-4faf-b323-2a213450d04f","Type":"ContainerDied","Data":"f061fba169c1b3f377c6add1a6159078f62ea49cea225c86a5c92811a58b3973"}
Jul 01 17:12:27.144643 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.144520    2556 scope.go:117] "RemoveContainer" containerID="500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f"
Jul 01 17:12:27.153033 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.153016    2556 scope.go:117] "RemoveContainer" containerID="bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d"
Jul 01 17:12:27.160160 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.160137    2556 scope.go:117] "RemoveContainer" containerID="22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9"
Jul 01 17:12:27.167619 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.167601    2556 scope.go:117] "RemoveContainer" containerID="500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f"
Jul 01 17:12:27.167881 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:27.167863    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f\": container with ID starting with 500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f not found: ID does not exist" containerID="500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f"
Jul 01 17:12:27.167921 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.167891    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f"} err="failed to get container status \"500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f\": rpc error: code = NotFound desc = could not find container \"500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f\": container with ID starting with 500337e3536d1639fae0a05e72ffe8e92da0408a20c298d00a23d87fd3696e4f not found: ID does not exist"
Jul 01 17:12:27.167921 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.167912    2556 scope.go:117] "RemoveContainer" containerID="bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d"
Jul 01 17:12:27.168146 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:27.168131    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d\": container with ID starting with bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d not found: ID does not exist" containerID="bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d"
Jul 01 17:12:27.168191 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.168150    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d"} err="failed to get container status \"bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d\": rpc error: code = NotFound desc = could not find container \"bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d\": container with ID starting with bcf4754bb37fbc74c7bc5092c9bda8f2c44468f669e902a43fdf62b47affd90d not found: ID does not exist"
Jul 01 17:12:27.168191 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.168162    2556 scope.go:117] "RemoveContainer" containerID="22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9"
Jul 01 17:12:27.168410 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:27.168393    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9\": container with ID starting with 22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9 not found: ID does not exist" containerID="22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9"
Jul 01 17:12:27.168445 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.168420    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9"} err="failed to get container status \"22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9\": rpc error: code = NotFound desc = could not find container \"22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9\": container with ID starting with 22d8b0f9dc0bb9ec8f57b3e39a851e3eedc565a6a9701428bb5f78f160c726e9 not found: ID does not exist"
Jul 01 17:12:27.168445 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.168396    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"]
Jul 01 17:12:27.173838 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.173816    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-89809-predictor-67f44f8c89-kfbbv"]
Jul 01 17:12:27.705077 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.705021    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 01 17:12:27.950618 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:27.950595    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:12:28.074139 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.074104    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" path="/var/lib/kubelet/pods/dd5a468c-c5ff-4faf-b323-2a213450d04f/volumes"
Jul 01 17:12:28.082076 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082040    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-rbac-proxy-sar-config\") pod \"f4c0c9d5-258b-4777-a905-83df72e038f2\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") "
Jul 01 17:12:28.082211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082092    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c9d5-258b-4777-a905-83df72e038f2-kserve-provision-location\") pod \"f4c0c9d5-258b-4777-a905-83df72e038f2\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") "
Jul 01 17:12:28.082211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082116    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls\") pod \"f4c0c9d5-258b-4777-a905-83df72e038f2\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") "
Jul 01 17:12:28.082211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082150    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5ps8\" (UniqueName: \"kubernetes.io/projected/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-api-access-v5ps8\") pod \"f4c0c9d5-258b-4777-a905-83df72e038f2\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") "
Jul 01 17:12:28.082211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082198    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-cabundle-cert\") pod \"f4c0c9d5-258b-4777-a905-83df72e038f2\" (UID: \"f4c0c9d5-258b-4777-a905-83df72e038f2\") "
Jul 01 17:12:28.082411 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082369    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4c0c9d5-258b-4777-a905-83df72e038f2" (UID: "f4c0c9d5-258b-4777-a905-83df72e038f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:12:28.082411 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082392    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c0c9d5-258b-4777-a905-83df72e038f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4c0c9d5-258b-4777-a905-83df72e038f2" (UID: "f4c0c9d5-258b-4777-a905-83df72e038f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:12:28.082566 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.082549    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4c0c9d5-258b-4777-a905-83df72e038f2" (UID: "f4c0c9d5-258b-4777-a905-83df72e038f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:12:28.084206 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.084188    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4c0c9d5-258b-4777-a905-83df72e038f2" (UID: "f4c0c9d5-258b-4777-a905-83df72e038f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:12:28.084305 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.084287    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-api-access-v5ps8" (OuterVolumeSpecName: "kube-api-access-v5ps8") pod "f4c0c9d5-258b-4777-a905-83df72e038f2" (UID: "f4c0c9d5-258b-4777-a905-83df72e038f2"). InnerVolumeSpecName "kube-api-access-v5ps8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:12:28.150914 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.150880    2556 generic.go:356] "Generic (PLEG): container finished" podID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerID="0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a" exitCode=0
Jul 01 17:12:28.151094 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.150958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerDied","Data":"0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a"}
Jul 01 17:12:28.152393 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.152371    2556 generic.go:356] "Generic (PLEG): container finished" podID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerID="505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e" exitCode=0
Jul 01 17:12:28.152463 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.152438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerDied","Data":"505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e"}
Jul 01 17:12:28.154415 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.154375    2556 generic.go:356] "Generic (PLEG): container finished" podID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerID="66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683" exitCode=0
Jul 01 17:12:28.154415 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.154405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerDied","Data":"66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683"}
Jul 01 17:12:28.154566 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.154453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj" event={"ID":"f4c0c9d5-258b-4777-a905-83df72e038f2","Type":"ContainerDied","Data":"fb6fb29f1032206a625ad5d02290da696432e1d1ec37ea00cdb0352c819f891f"}
Jul 01 17:12:28.154566 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.154466    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"
Jul 01 17:12:28.154566 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.154481    2556 scope.go:117] "RemoveContainer" containerID="22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8"
Jul 01 17:12:28.163633 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.163612    2556 scope.go:117] "RemoveContainer" containerID="66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683"
Jul 01 17:12:28.173091 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.173054    2556 scope.go:117] "RemoveContainer" containerID="179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933"
Jul 01 17:12:28.182991 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.182965    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:28.183096 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.183000    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:28.183096 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.183016    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4c0c9d5-258b-4777-a905-83df72e038f2-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:28.183096 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.183033    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4c0c9d5-258b-4777-a905-83df72e038f2-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:28.183096 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.183047    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5ps8\" (UniqueName: \"kubernetes.io/projected/f4c0c9d5-258b-4777-a905-83df72e038f2-kube-api-access-v5ps8\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:12:28.183772 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.183676    2556 scope.go:117] "RemoveContainer" containerID="22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8"
Jul 01 17:12:28.184357 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:28.184334    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8\": container with ID starting with 22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8 not found: ID does not exist" containerID="22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8"
Jul 01 17:12:28.184463 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.184367    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8"} err="failed to get container status \"22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8\": rpc error: code = NotFound desc = could not find container \"22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8\": container with ID starting with 22230ce1c4ba71882953dfbe7d9cf6063ebdc1d33d10d7869fe401849fdefbe8 not found: ID does not exist"
Jul 01 17:12:28.184463 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.184390    2556 scope.go:117] "RemoveContainer" containerID="66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683"
Jul 01 17:12:28.184772 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:28.184750    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683\": container with ID starting with 66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683 not found: ID does not exist" containerID="66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683"
Jul 01 17:12:28.184828 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.184782    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683"} err="failed to get container status \"66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683\": rpc error: code = NotFound desc = could not find container \"66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683\": container with ID starting with 66467ccdf054192956ea2083bf131bc1ad6b91ff3e0ae1033423a04e5433a683 not found: ID does not exist"
Jul 01 17:12:28.184828 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.184804    2556 scope.go:117] "RemoveContainer" containerID="179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933"
Jul 01 17:12:28.185396 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:12:28.185375    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933\": container with ID starting with 179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933 not found: ID does not exist" containerID="179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933"
Jul 01 17:12:28.185450 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.185402    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933"} err="failed to get container status \"179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933\": rpc error: code = NotFound desc = could not find container \"179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933\": container with ID starting with 179bfdc3164b63bba9ebc927eb03e8a0739014e31fb6623aebedf8c3e11fb933 not found: ID does not exist"
Jul 01 17:12:28.209213 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.209189    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"]
Jul 01 17:12:28.219306 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:28.219286    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-89809-predictor-955cb8778-jgldj"]
Jul 01 17:12:29.161283 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.161245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerStarted","Data":"7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312"}
Jul 01 17:12:29.161718 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.161354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerStarted","Data":"d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9"}
Jul 01 17:12:29.161718 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.161601    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:29.161718 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.161657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:29.163437 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.163410    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerStarted","Data":"7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689"}
Jul 01 17:12:29.163559 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.163428    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:12:29.163559 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.163458    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerStarted","Data":"265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283"}
Jul 01 17:12:29.163672 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.163626    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:29.185112 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.185044    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podStartSLOduration=2.544245869 podStartE2EDuration="6.185030768s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.640784899s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:12:29.183680552 +0000 UTC m=+839.526021212" watchObservedRunningTime="2026-07-01 17:12:29.185030768 +0000 UTC m=+839.527371427"
Jul 01 17:12:29.205829 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:29.205770    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podStartSLOduration=2.57157793 podStartE2EDuration="6.205751801s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.634173871s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:12:29.205417027 +0000 UTC m=+839.547757687" watchObservedRunningTime="2026-07-01 17:12:29.205751801 +0000 UTC m=+839.548092462"
Jul 01 17:12:30.072685 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:30.072660    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" path="/var/lib/kubelet/pods/f4c0c9d5-258b-4777-a905-83df72e038f2/volumes"
Jul 01 17:12:30.166805 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:30.166769    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:12:30.167186 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:30.166884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:30.168075 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:30.168029    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:12:31.170759 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:31.170718    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:12:35.171677 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:35.171647    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:12:35.172230 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:35.172203    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:12:36.175339 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:36.175312    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:12:36.175944 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:36.175915    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:12:45.172333 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:45.172291    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:12:46.176449 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:46.176408    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:12:55.173092 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:55.173035    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:12:56.176046 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:12:56.176007    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:13:05.172311 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:05.172268    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:13:06.176430 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:06.176388    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:13:15.173188 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:15.173150    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:13:16.176132 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:16.176090    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:13:25.172729 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:25.172645    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:13:26.175875 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:26.175840    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:13:30.613930 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:30.613904    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:13:30.615190 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:30.615169    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:13:35.172349 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:35.172303    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:13:36.176733 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:36.176705    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:13:45.172769 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:13:45.172736    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:14:13.408603 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408567    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"]
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408838    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="storage-initializer"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408849    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="storage-initializer"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408859    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kube-rbac-proxy"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408866    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kube-rbac-proxy"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408881    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="storage-initializer"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408887    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="storage-initializer"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408896    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408902    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408907    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408912    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408918    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kube-rbac-proxy"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408926    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kube-rbac-proxy"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408965    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kube-rbac-proxy"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408972    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kserve-container"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408978    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd5a468c-c5ff-4faf-b323-2a213450d04f" containerName="kube-rbac-proxy"
Jul 01 17:14:13.409081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.408984    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c0c9d5-258b-4777-a905-83df72e038f2" containerName="kserve-container"
Jul 01 17:14:13.411824 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.411808    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.414561 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.414540    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-df9a9-kube-rbac-proxy-sar-config\""
Jul 01 17:14:13.414684 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.414536    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-df9a9-predictor-serving-cert\""
Jul 01 17:14:13.423633 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.423608    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"]
Jul 01 17:14:13.463391 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.463362    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"]
Jul 01 17:14:13.463666 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.463641    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" containerID="cri-o://d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9" gracePeriod=30
Jul 01 17:14:13.463778 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.463657    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kube-rbac-proxy" containerID="cri-o://7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312" gracePeriod=30
Jul 01 17:14:13.499998 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.499965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2j27\" (UniqueName: \"kubernetes.io/projected/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-api-access-w2j27\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.500179 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.500037    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.500179 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.500079    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.511236 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.511206    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"]
Jul 01 17:14:13.511462 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.511435    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" containerID="cri-o://265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283" gracePeriod=30
Jul 01 17:14:13.511533 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.511500    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kube-rbac-proxy" containerID="cri-o://7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689" gracePeriod=30
Jul 01 17:14:13.600877 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.600831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2j27\" (UniqueName: \"kubernetes.io/projected/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-api-access-w2j27\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.601049 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.600899    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.601049 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.600936    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.601049 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:13.601001    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-serving-cert: secret "message-dumper-raw-df9a9-predictor-serving-cert" not found
Jul 01 17:14:13.601195 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:13.601097    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls podName:e3b1cb1a-3439-48a1-90a7-245f435e8c54 nodeName:}" failed. No retries permitted until 2026-07-01 17:14:14.101046044 +0000 UTC m=+944.443386720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls") pod "message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" (UID: "e3b1cb1a-3439-48a1-90a7-245f435e8c54") : secret "message-dumper-raw-df9a9-predictor-serving-cert" not found
Jul 01 17:14:13.601584 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.601556    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:13.611662 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:13.611637    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2j27\" (UniqueName: \"kubernetes.io/projected/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-api-access-w2j27\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:14.105205 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.105163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:14.107534 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.107510    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls\") pod \"message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") " pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:14.322226 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.322189    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:14.455208 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.455182    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"]
Jul 01 17:14:14.456937 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:14:14.456915    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b1cb1a_3439_48a1_90a7_245f435e8c54.slice/crio-317f295d1f2140bd9a548010a145635f164fc10d2f9fab12add28c6baf8b6813 WatchSource:0}: Error finding container 317f295d1f2140bd9a548010a145635f164fc10d2f9fab12add28c6baf8b6813: Status 404 returned error can't find the container with id 317f295d1f2140bd9a548010a145635f164fc10d2f9fab12add28c6baf8b6813
Jul 01 17:14:14.458703 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.458687    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:14:14.555100 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.555048    2556 generic.go:356] "Generic (PLEG): container finished" podID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerID="7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689" exitCode=2
Jul 01 17:14:14.555266 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.555123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerDied","Data":"7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689"}
Jul 01 17:14:14.556321 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.556298    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" event={"ID":"e3b1cb1a-3439-48a1-90a7-245f435e8c54","Type":"ContainerStarted","Data":"317f295d1f2140bd9a548010a145635f164fc10d2f9fab12add28c6baf8b6813"}
Jul 01 17:14:14.558213 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.558193    2556 generic.go:356] "Generic (PLEG): container finished" podID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerID="7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312" exitCode=2
Jul 01 17:14:14.558299 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:14.558220    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerDied","Data":"7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312"}
Jul 01 17:14:15.167758 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:15.167703    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 01 17:14:15.172536 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:15.172509    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 01 17:14:16.171272 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.171184    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 01 17:14:16.176475 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.176450    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 01 17:14:16.568197 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.568164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" event={"ID":"e3b1cb1a-3439-48a1-90a7-245f435e8c54","Type":"ContainerStarted","Data":"54c2d07785d364d23a6ee816b4ddbb38ed687d04ea4f5f57b08926a62412d3e3"}
Jul 01 17:14:16.568366 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.568227    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" event={"ID":"e3b1cb1a-3439-48a1-90a7-245f435e8c54","Type":"ContainerStarted","Data":"5c2ce0266e006ea0a398d1a9534166e30f90ba86515ea16bc22df9d2c8e71c80"}
Jul 01 17:14:16.568404 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.568376    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:16.568537 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.568524    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:16.570221 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.570203    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:16.589874 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:16.589826    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" podStartSLOduration=2.271674437 podStartE2EDuration="3.589811702s" totalImagesPullingTime="1.318137265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:14:16.588679347 +0000 UTC m=+946.931020008" watchObservedRunningTime="2026-07-01 17:14:16.589811702 +0000 UTC m=+946.932152371"
Jul 01 17:14:17.460161 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.460139    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:14:17.573861 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.573776    2556 generic.go:356] "Generic (PLEG): container finished" podID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerID="265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283" exitCode=0
Jul 01 17:14:17.573861 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.573829    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"
Jul 01 17:14:17.574040 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.573863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerDied","Data":"265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283"}
Jul 01 17:14:17.574040 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.573931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm" event={"ID":"ec6d77c8-9be9-404e-8a0f-00503f939b4d","Type":"ContainerDied","Data":"44124234e835b83789985b19f9cdeacb7423872ebe191a00e29b5373ed3bdad6"}
Jul 01 17:14:17.574040 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.573960    2556 scope.go:117] "RemoveContainer" containerID="7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689"
Jul 01 17:14:17.581905 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.581886    2556 scope.go:117] "RemoveContainer" containerID="265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283"
Jul 01 17:14:17.588883 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.588865    2556 scope.go:117] "RemoveContainer" containerID="505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e"
Jul 01 17:14:17.595743 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.595717    2556 scope.go:117] "RemoveContainer" containerID="7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689"
Jul 01 17:14:17.595990 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:17.595971    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689\": container with ID starting with 7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689 not found: ID does not exist" containerID="7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689"
Jul 01 17:14:17.596036 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.596001    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689"} err="failed to get container status \"7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689\": rpc error: code = NotFound desc = could not find container \"7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689\": container with ID starting with 7cdd7aab85a947f1c2dd195bd9e0121bfbb5d603c65f7c1a6a36861312a2f689 not found: ID does not exist"
Jul 01 17:14:17.596036 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.596022    2556 scope.go:117] "RemoveContainer" containerID="265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283"
Jul 01 17:14:17.596281 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:17.596266    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283\": container with ID starting with 265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283 not found: ID does not exist" containerID="265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283"
Jul 01 17:14:17.596321 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.596285    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283"} err="failed to get container status \"265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283\": rpc error: code = NotFound desc = could not find container \"265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283\": container with ID starting with 265490f4193eccdafc0be87d9c0bed81212715f9d50193a058f35ee015568283 not found: ID does not exist"
Jul 01 17:14:17.596321 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.596297    2556 scope.go:117] "RemoveContainer" containerID="505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e"
Jul 01 17:14:17.596508 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:17.596494    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e\": container with ID starting with 505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e not found: ID does not exist" containerID="505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e"
Jul 01 17:14:17.596546 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.596511    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e"} err="failed to get container status \"505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e\": rpc error: code = NotFound desc = could not find container \"505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e\": container with ID starting with 505db61a565d07c133378a04d2f05510fbab64589f6861b974f07c47ea24798e not found: ID does not exist"
Jul 01 17:14:17.637838 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.637800    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls\") pod \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") "
Jul 01 17:14:17.637973 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.637847    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5qjd\" (UniqueName: \"kubernetes.io/projected/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-api-access-d5qjd\") pod \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") "
Jul 01 17:14:17.637973 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.637892    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-rbac-proxy-sar-config\") pod \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") "
Jul 01 17:14:17.638045 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.638012    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-cabundle-cert\") pod \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") "
Jul 01 17:14:17.638145 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.638124    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kserve-provision-location\") pod \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\" (UID: \"ec6d77c8-9be9-404e-8a0f-00503f939b4d\") "
Jul 01 17:14:17.638304 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.638275    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec6d77c8-9be9-404e-8a0f-00503f939b4d" (UID: "ec6d77c8-9be9-404e-8a0f-00503f939b4d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:14:17.638446 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.638426    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec6d77c8-9be9-404e-8a0f-00503f939b4d" (UID: "ec6d77c8-9be9-404e-8a0f-00503f939b4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:14:17.638509 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.638460    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec6d77c8-9be9-404e-8a0f-00503f939b4d" (UID: "ec6d77c8-9be9-404e-8a0f-00503f939b4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:14:17.638509 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.638487    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:17.640098 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.640053    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-api-access-d5qjd" (OuterVolumeSpecName: "kube-api-access-d5qjd") pod "ec6d77c8-9be9-404e-8a0f-00503f939b4d" (UID: "ec6d77c8-9be9-404e-8a0f-00503f939b4d"). InnerVolumeSpecName "kube-api-access-d5qjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:14:17.640098 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.640074    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec6d77c8-9be9-404e-8a0f-00503f939b4d" (UID: "ec6d77c8-9be9-404e-8a0f-00503f939b4d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:14:17.739832 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.739793    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qjd\" (UniqueName: \"kubernetes.io/projected/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kube-api-access-d5qjd\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:17.739832 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.739825    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec6d77c8-9be9-404e-8a0f-00503f939b4d-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:17.739832 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.739837    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec6d77c8-9be9-404e-8a0f-00503f939b4d-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:17.739832 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.739845    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec6d77c8-9be9-404e-8a0f-00503f939b4d-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:17.896688 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.896660    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"]
Jul 01 17:14:17.903000 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:17.902970    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-e08af-predictor-66b5fcf89c-tvfqm"]
Jul 01 17:14:18.072443 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.072415    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" path="/var/lib/kubelet/pods/ec6d77c8-9be9-404e-8a0f-00503f939b4d/volumes"
Jul 01 17:14:18.514153 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.514129    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:14:18.580019 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.579986    2556 generic.go:356] "Generic (PLEG): container finished" podID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerID="d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9" exitCode=0
Jul 01 17:14:18.580216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.580040    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"
Jul 01 17:14:18.580216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.580085    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerDied","Data":"d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9"}
Jul 01 17:14:18.580216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.580144    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp" event={"ID":"c5a40b5d-a681-4b80-942b-dd06dc63769f","Type":"ContainerDied","Data":"e3e3e0668c0a19e383be357a0fbd38c2c4db5e4e5747d6da7dd5633f37ab30df"}
Jul 01 17:14:18.580216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.580174    2556 scope.go:117] "RemoveContainer" containerID="7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312"
Jul 01 17:14:18.587672 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.587652    2556 scope.go:117] "RemoveContainer" containerID="d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9"
Jul 01 17:14:18.594510 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.594493    2556 scope.go:117] "RemoveContainer" containerID="0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a"
Jul 01 17:14:18.602521 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.602506    2556 scope.go:117] "RemoveContainer" containerID="7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312"
Jul 01 17:14:18.602836 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:18.602807    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312\": container with ID starting with 7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312 not found: ID does not exist" containerID="7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312"
Jul 01 17:14:18.602936 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.602840    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312"} err="failed to get container status \"7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312\": rpc error: code = NotFound desc = could not find container \"7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312\": container with ID starting with 7cd3c8185f6c9dfd0bce6832bbf4b9a8900eee128898cb65064cb034e1740312 not found: ID does not exist"
Jul 01 17:14:18.602936 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.602859    2556 scope.go:117] "RemoveContainer" containerID="d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9"
Jul 01 17:14:18.603108 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:18.603093    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9\": container with ID starting with d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9 not found: ID does not exist" containerID="d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9"
Jul 01 17:14:18.603159 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.603112    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9"} err="failed to get container status \"d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9\": rpc error: code = NotFound desc = could not find container \"d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9\": container with ID starting with d8ffa15f707f8f7b92600b5c9a080e66687404a4eb7d564c69bf6fbfcfe229b9 not found: ID does not exist"
Jul 01 17:14:18.603159 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.603124    2556 scope.go:117] "RemoveContainer" containerID="0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a"
Jul 01 17:14:18.603348 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:18.603331    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a\": container with ID starting with 0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a not found: ID does not exist" containerID="0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a"
Jul 01 17:14:18.603403 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.603356    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a"} err="failed to get container status \"0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a\": rpc error: code = NotFound desc = could not find container \"0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a\": container with ID starting with 0f762d6b2d5cfff45ecab6e37ddf32b9abc241643b2a518c7e08a57633cbfa0a not found: ID does not exist"
Jul 01 17:14:18.646948 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.646914    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls\") pod \"c5a40b5d-a681-4b80-942b-dd06dc63769f\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") "
Jul 01 17:14:18.647112 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.646965    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tnl5\" (UniqueName: \"kubernetes.io/projected/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-api-access-8tnl5\") pod \"c5a40b5d-a681-4b80-942b-dd06dc63769f\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") "
Jul 01 17:14:18.647112 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.646986    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a40b5d-a681-4b80-942b-dd06dc63769f-kserve-provision-location\") pod \"c5a40b5d-a681-4b80-942b-dd06dc63769f\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") "
Jul 01 17:14:18.647112 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.647053    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-rbac-proxy-sar-config\") pod \"c5a40b5d-a681-4b80-942b-dd06dc63769f\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") "
Jul 01 17:14:18.647292 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.647117    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-cabundle-cert\") pod \"c5a40b5d-a681-4b80-942b-dd06dc63769f\" (UID: \"c5a40b5d-a681-4b80-942b-dd06dc63769f\") "
Jul 01 17:14:18.647443 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.647352    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a40b5d-a681-4b80-942b-dd06dc63769f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5a40b5d-a681-4b80-942b-dd06dc63769f" (UID: "c5a40b5d-a681-4b80-942b-dd06dc63769f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:14:18.647519 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.647427    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5a40b5d-a681-4b80-942b-dd06dc63769f" (UID: "c5a40b5d-a681-4b80-942b-dd06dc63769f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:14:18.647519 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.647482    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5a40b5d-a681-4b80-942b-dd06dc63769f" (UID: "c5a40b5d-a681-4b80-942b-dd06dc63769f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:14:18.649110 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.649090    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5a40b5d-a681-4b80-942b-dd06dc63769f" (UID: "c5a40b5d-a681-4b80-942b-dd06dc63769f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:14:18.649283 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.649266    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-api-access-8tnl5" (OuterVolumeSpecName: "kube-api-access-8tnl5") pod "c5a40b5d-a681-4b80-942b-dd06dc63769f" (UID: "c5a40b5d-a681-4b80-942b-dd06dc63769f"). InnerVolumeSpecName "kube-api-access-8tnl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:14:18.748445 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.748406    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:18.748445 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.748444    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5a40b5d-a681-4b80-942b-dd06dc63769f-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:18.748643 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.748458    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tnl5\" (UniqueName: \"kubernetes.io/projected/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-api-access-8tnl5\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:18.748643 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.748471    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5a40b5d-a681-4b80-942b-dd06dc63769f-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:18.748643 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.748484    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5a40b5d-a681-4b80-942b-dd06dc63769f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:14:18.902443 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.902407    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"]
Jul 01 17:14:18.908823 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:18.908793    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e08af-predictor-76bf495994-9kvmp"]
Jul 01 17:14:20.073830 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:20.073797    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" path="/var/lib/kubelet/pods/c5a40b5d-a681-4b80-942b-dd06dc63769f/volumes"
Jul 01 17:14:23.585835 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:23.585808    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:14:33.439522 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439483    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"]
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439806    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="storage-initializer"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439820    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="storage-initializer"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439834    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439839    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439847    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kube-rbac-proxy"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439852    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kube-rbac-proxy"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439859    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="storage-initializer"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439864    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="storage-initializer"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439870    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439875    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439881    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kube-rbac-proxy"
Jul 01 17:14:33.439887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439886    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kube-rbac-proxy"
Jul 01 17:14:33.440274 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439930    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kube-rbac-proxy"
Jul 01 17:14:33.440274 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439936    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kserve-container"
Jul 01 17:14:33.440274 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439943    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec6d77c8-9be9-404e-8a0f-00503f939b4d" containerName="kserve-container"
Jul 01 17:14:33.440274 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.439948    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a40b5d-a681-4b80-942b-dd06dc63769f" containerName="kube-rbac-proxy"
Jul 01 17:14:33.444417 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.444399    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.446780 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.446756    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-df9a9-predictor-serving-cert\""
Jul 01 17:14:33.447005 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.446990    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:14:33.447169 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.447001    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-df9a9-kube-rbac-proxy-sar-config\""
Jul 01 17:14:33.447169 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.447040    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:14:33.448609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.448589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.448697 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.448615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-cabundle-cert\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.448697 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.448670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd62387-8aad-469a-ac5e-817e0e169f8b-kserve-provision-location\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.448797 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.448725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.448797 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.448762    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mwb\" (UniqueName: \"kubernetes.io/projected/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-api-access-k7mwb\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.453791 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.453770    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"]
Jul 01 17:14:33.549918 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.549877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd62387-8aad-469a-ac5e-817e0e169f8b-kserve-provision-location\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.549918 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.549923    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.550185 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.549944    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mwb\" (UniqueName: \"kubernetes.io/projected/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-api-access-k7mwb\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.550185 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.549977    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.550185 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:33.550047    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-serving-cert: secret "isvc-logger-raw-df9a9-predictor-serving-cert" not found
Jul 01 17:14:33.550185 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.550105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-cabundle-cert\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.550185 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:14:33.550142    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls podName:5cd62387-8aad-469a-ac5e-817e0e169f8b nodeName:}" failed. No retries permitted until 2026-07-01 17:14:34.050120091 +0000 UTC m=+964.392460729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls") pod "isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" (UID: "5cd62387-8aad-469a-ac5e-817e0e169f8b") : secret "isvc-logger-raw-df9a9-predictor-serving-cert" not found
Jul 01 17:14:33.550410 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.550375    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd62387-8aad-469a-ac5e-817e0e169f8b-kserve-provision-location\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.550807 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.550784    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-cabundle-cert\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.550845 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.550784    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:33.558466 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:33.558446    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mwb\" (UniqueName: \"kubernetes.io/projected/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-api-access-k7mwb\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:34.053605 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:34.053554    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:34.056070 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:34.056039    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls\") pod \"isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") " pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:34.355496 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:34.355403    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:34.488859 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:34.488827    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"]
Jul 01 17:14:34.491655 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:14:34.491632    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd62387_8aad_469a_ac5e_817e0e169f8b.slice/crio-c098a7f4b5233ed392ef3ca5d6d0c18c17d4e2697e53903dc4bebc0f925884ba WatchSource:0}: Error finding container c098a7f4b5233ed392ef3ca5d6d0c18c17d4e2697e53903dc4bebc0f925884ba: Status 404 returned error can't find the container with id c098a7f4b5233ed392ef3ca5d6d0c18c17d4e2697e53903dc4bebc0f925884ba
Jul 01 17:14:34.637095 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:34.636996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerStarted","Data":"ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420"}
Jul 01 17:14:34.637234 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:34.637091    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerStarted","Data":"c098a7f4b5233ed392ef3ca5d6d0c18c17d4e2697e53903dc4bebc0f925884ba"}
Jul 01 17:14:38.653054 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:38.652973    2556 generic.go:356] "Generic (PLEG): container finished" podID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerID="ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420" exitCode=0
Jul 01 17:14:38.653054 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:38.653034    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerDied","Data":"ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420"}
Jul 01 17:14:39.659696 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.659663    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerStarted","Data":"cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f"}
Jul 01 17:14:39.660146 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.659742    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerStarted","Data":"15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c"}
Jul 01 17:14:39.660146 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.659770    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerStarted","Data":"c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c"}
Jul 01 17:14:39.660146 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.660037    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:39.660146 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.660114    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:39.660146 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.660149    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:39.661585 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.661551    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:14:39.662276 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.662249    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:14:39.684435 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:39.684385    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podStartSLOduration=2.9037271860000002 podStartE2EDuration="6.684372145s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780644959s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:14:39.682461185 +0000 UTC m=+970.024801846" watchObservedRunningTime="2026-07-01 17:14:39.684372145 +0000 UTC m=+970.026712805"
Jul 01 17:14:40.663373 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:40.663327    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:14:40.663768 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:40.663706    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:14:45.667614 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:45.667578    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:14:45.668301 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:45.668249    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:14:45.668504 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:45.668480    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:14:55.668717 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:55.668677    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:14:55.669214 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:14:55.669054    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:15:05.668509 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:05.668458    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:15:05.668992 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:05.668892    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:15:15.668987 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:15.668944    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:15:15.669521 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:15.669426    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:15:25.668757 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:25.668713    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:15:25.669341 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:25.669225    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:15:35.668329 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:35.668278    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:15:35.668874 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:35.668720    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:15:45.668603 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:45.668560    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:15:45.669098 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:45.669079    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:15:54.073532 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:54.073505    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:15:54.074127 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:15:54.073617    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:16:08.457003 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.456970    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r_e3b1cb1a-3439-48a1-90a7-245f435e8c54/kserve-container/0.log"
Jul 01 17:16:08.778372 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.778335    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"]
Jul 01 17:16:08.778789 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.778759    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" containerID="cri-o://cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f" gracePeriod=30
Jul 01 17:16:08.779006 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.778772    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy" containerID="cri-o://15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c" gracePeriod=30
Jul 01 17:16:08.779836 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.779380    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" containerID="cri-o://c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c" gracePeriod=30
Jul 01 17:16:08.802307 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.802280    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"]
Jul 01 17:16:08.805912 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.805893    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.808588 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.808566    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-76ae4-predictor-serving-cert\""
Jul 01 17:16:08.808666 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.808585    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-76ae4-kube-rbac-proxy-sar-config\""
Jul 01 17:16:08.820996 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.820971    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"]
Jul 01 17:16:08.862517 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.862490    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"]
Jul 01 17:16:08.862725 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.862701    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kserve-container" containerID="cri-o://5c2ce0266e006ea0a398d1a9534166e30f90ba86515ea16bc22df9d2c8e71c80" gracePeriod=30
Jul 01 17:16:08.862809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.862779    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kube-rbac-proxy" containerID="cri-o://54c2d07785d364d23a6ee816b4ddbb38ed687d04ea4f5f57b08926a62412d3e3" gracePeriod=30
Jul 01 17:16:08.867794 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.867745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.867794 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.867785    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.867927 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.867818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-cabundle-cert\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.867927 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.867872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.867927 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.867914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gtg\" (UniqueName: \"kubernetes.io/projected/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-api-access-h2gtg\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.968437 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.968407    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.968558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.968452    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.968558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.968481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-cabundle-cert\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.968558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.968542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.968720 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:16:08.968565    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-serving-cert: secret "isvc-sklearn-scale-raw-76ae4-predictor-serving-cert" not found
Jul 01 17:16:08.968720 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.968589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gtg\" (UniqueName: \"kubernetes.io/projected/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-api-access-h2gtg\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.968720 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:16:08.968633    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls podName:9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:09.468612337 +0000 UTC m=+1059.810952979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls") pod "isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" (UID: "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9") : secret "isvc-sklearn-scale-raw-76ae4-predictor-serving-cert" not found
Jul 01 17:16:08.969052 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.969027    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.969254 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.969236    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-cabundle-cert\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.969318 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.969259    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.980544 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.980514    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gtg\" (UniqueName: \"kubernetes.io/projected/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-api-access-h2gtg\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:08.983442 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.983417    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerID="54c2d07785d364d23a6ee816b4ddbb38ed687d04ea4f5f57b08926a62412d3e3" exitCode=2
Jul 01 17:16:08.983442 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.983441    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerID="5c2ce0266e006ea0a398d1a9534166e30f90ba86515ea16bc22df9d2c8e71c80" exitCode=2
Jul 01 17:16:08.983598 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.983452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" event={"ID":"e3b1cb1a-3439-48a1-90a7-245f435e8c54","Type":"ContainerDied","Data":"54c2d07785d364d23a6ee816b4ddbb38ed687d04ea4f5f57b08926a62412d3e3"}
Jul 01 17:16:08.983598 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.983520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" event={"ID":"e3b1cb1a-3439-48a1-90a7-245f435e8c54","Type":"ContainerDied","Data":"5c2ce0266e006ea0a398d1a9534166e30f90ba86515ea16bc22df9d2c8e71c80"}
Jul 01 17:16:08.986324 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.986278    2556 generic.go:356] "Generic (PLEG): container finished" podID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerID="15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c" exitCode=2
Jul 01 17:16:08.986324 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:08.986315    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerDied","Data":"15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c"}
Jul 01 17:16:09.104043 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.104017    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:16:09.170723 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.170696    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-rbac-proxy-sar-config\") pod \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") "
Jul 01 17:16:09.170891 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.170739    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2j27\" (UniqueName: \"kubernetes.io/projected/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-api-access-w2j27\") pod \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") "
Jul 01 17:16:09.170891 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.170760    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls\") pod \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\" (UID: \"e3b1cb1a-3439-48a1-90a7-245f435e8c54\") "
Jul 01 17:16:09.171092 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.171045    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3b1cb1a-3439-48a1-90a7-245f435e8c54" (UID: "e3b1cb1a-3439-48a1-90a7-245f435e8c54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:16:09.172776 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.172755    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3b1cb1a-3439-48a1-90a7-245f435e8c54" (UID: "e3b1cb1a-3439-48a1-90a7-245f435e8c54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:16:09.172840 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.172825    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-api-access-w2j27" (OuterVolumeSpecName: "kube-api-access-w2j27") pod "e3b1cb1a-3439-48a1-90a7-245f435e8c54" (UID: "e3b1cb1a-3439-48a1-90a7-245f435e8c54"). InnerVolumeSpecName "kube-api-access-w2j27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:16:09.271436 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.271391    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:09.271436 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.271430    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2j27\" (UniqueName: \"kubernetes.io/projected/e3b1cb1a-3439-48a1-90a7-245f435e8c54-kube-api-access-w2j27\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:09.271436 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.271439    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3b1cb1a-3439-48a1-90a7-245f435e8c54-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:09.472970 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.472880    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:09.475267 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.475249    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls\") pod \"isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:09.715540 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.715488    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:09.850295 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.850168    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"]
Jul 01 17:16:09.852941 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:16:09.852915    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f04d4e1_4be5_45b9_a6d3_e69a2274cdf9.slice/crio-fad0926a930f966508812fb413850028eefc1a5d4db81356b365c84aa2dee971 WatchSource:0}: Error finding container fad0926a930f966508812fb413850028eefc1a5d4db81356b365c84aa2dee971: Status 404 returned error can't find the container with id fad0926a930f966508812fb413850028eefc1a5d4db81356b365c84aa2dee971
Jul 01 17:16:09.992429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.992400    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerStarted","Data":"db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343"}
Jul 01 17:16:09.992613 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.992464    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerStarted","Data":"fad0926a930f966508812fb413850028eefc1a5d4db81356b365c84aa2dee971"}
Jul 01 17:16:09.994020 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.993997    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r" event={"ID":"e3b1cb1a-3439-48a1-90a7-245f435e8c54","Type":"ContainerDied","Data":"317f295d1f2140bd9a548010a145635f164fc10d2f9fab12add28c6baf8b6813"}
Jul 01 17:16:09.994158 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.994051    2556 scope.go:117] "RemoveContainer" containerID="54c2d07785d364d23a6ee816b4ddbb38ed687d04ea4f5f57b08926a62412d3e3"
Jul 01 17:16:09.994158 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:09.994001    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"
Jul 01 17:16:10.001987 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:10.001962    2556 scope.go:117] "RemoveContainer" containerID="5c2ce0266e006ea0a398d1a9534166e30f90ba86515ea16bc22df9d2c8e71c80"
Jul 01 17:16:10.042579 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:10.042548    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"]
Jul 01 17:16:10.045836 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:10.045811    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-df9a9-predictor-d9498cb4c-g7h6r"]
Jul 01 17:16:10.073572 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:10.073527    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" path="/var/lib/kubelet/pods/e3b1cb1a-3439-48a1-90a7-245f435e8c54/volumes"
Jul 01 17:16:10.663734 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:10.663692    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 17:16:14.011333 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:14.011245    2556 generic.go:356] "Generic (PLEG): container finished" podID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerID="c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c" exitCode=0
Jul 01 17:16:14.011333 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:14.011313    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerDied","Data":"c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c"}
Jul 01 17:16:14.012550 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:14.012530    2556 generic.go:356] "Generic (PLEG): container finished" podID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerID="db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343" exitCode=0
Jul 01 17:16:14.012631 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:14.012599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerDied","Data":"db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343"}
Jul 01 17:16:14.069247 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:14.069207    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:16:14.069725 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:14.069691    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:16:15.018116 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:15.018056    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerStarted","Data":"6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276"}
Jul 01 17:16:15.018609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:15.018153    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerStarted","Data":"3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84"}
Jul 01 17:16:15.018609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:15.018366    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:15.039775 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:15.039720    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podStartSLOduration=3.213226467 podStartE2EDuration="7.039702985s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.826476518s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:15.038018033 +0000 UTC m=+1065.380358733" watchObservedRunningTime="2026-07-01 17:16:15.039702985 +0000 UTC m=+1065.382043645"
Jul 01 17:16:15.663991 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:15.663946    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 17:16:16.021640 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:16.021611    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:16.023082 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:16.023036    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:16:17.025915 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:17.025870    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:16:20.663623 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:20.663577    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 17:16:20.664096 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:20.663720    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:16:22.030782 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:22.030703    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:16:22.031297 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:22.031221    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:16:24.068929 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:24.068881    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:16:24.069355 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:24.069022    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:16:25.664407 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:25.664365    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 17:16:30.664403 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:30.664357    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 17:16:32.032181 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:32.032144    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:16:34.068774 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:34.068730    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 01 17:16:34.069227 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:34.069106    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:16:34.073456 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:34.073439    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:16:34.073556 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:34.073529    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:16:35.663468 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:35.663422    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 01 17:16:38.974698 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:38.974676    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:16:39.094216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094124    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7mwb\" (UniqueName: \"kubernetes.io/projected/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-api-access-k7mwb\") pod \"5cd62387-8aad-469a-ac5e-817e0e169f8b\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") "
Jul 01 17:16:39.094216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094166    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls\") pod \"5cd62387-8aad-469a-ac5e-817e0e169f8b\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") "
Jul 01 17:16:39.094216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094191    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd62387-8aad-469a-ac5e-817e0e169f8b-kserve-provision-location\") pod \"5cd62387-8aad-469a-ac5e-817e0e169f8b\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") "
Jul 01 17:16:39.094216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094214    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-rbac-proxy-sar-config\") pod \"5cd62387-8aad-469a-ac5e-817e0e169f8b\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") "
Jul 01 17:16:39.094547 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094286    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-cabundle-cert\") pod \"5cd62387-8aad-469a-ac5e-817e0e169f8b\" (UID: \"5cd62387-8aad-469a-ac5e-817e0e169f8b\") "
Jul 01 17:16:39.094547 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094478    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd62387-8aad-469a-ac5e-817e0e169f8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cd62387-8aad-469a-ac5e-817e0e169f8b" (UID: "5cd62387-8aad-469a-ac5e-817e0e169f8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:16:39.094660 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094584    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cd62387-8aad-469a-ac5e-817e0e169f8b" (UID: "5cd62387-8aad-469a-ac5e-817e0e169f8b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:16:39.094714 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.094651    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cd62387-8aad-469a-ac5e-817e0e169f8b" (UID: "5cd62387-8aad-469a-ac5e-817e0e169f8b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:16:39.096391 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.096370    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-api-access-k7mwb" (OuterVolumeSpecName: "kube-api-access-k7mwb") pod "5cd62387-8aad-469a-ac5e-817e0e169f8b" (UID: "5cd62387-8aad-469a-ac5e-817e0e169f8b"). InnerVolumeSpecName "kube-api-access-k7mwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:16:39.096473 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.096397    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cd62387-8aad-469a-ac5e-817e0e169f8b" (UID: "5cd62387-8aad-469a-ac5e-817e0e169f8b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:16:39.107412 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.107384    2556 generic.go:356] "Generic (PLEG): container finished" podID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerID="cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f" exitCode=137
Jul 01 17:16:39.107528 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.107425    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerDied","Data":"cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f"}
Jul 01 17:16:39.107528 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.107450    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"
Jul 01 17:16:39.107528 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.107483    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l" event={"ID":"5cd62387-8aad-469a-ac5e-817e0e169f8b","Type":"ContainerDied","Data":"c098a7f4b5233ed392ef3ca5d6d0c18c17d4e2697e53903dc4bebc0f925884ba"}
Jul 01 17:16:39.107528 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.107512    2556 scope.go:117] "RemoveContainer" containerID="cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f"
Jul 01 17:16:39.115729 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.115611    2556 scope.go:117] "RemoveContainer" containerID="15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c"
Jul 01 17:16:39.122616 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.122601    2556 scope.go:117] "RemoveContainer" containerID="c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c"
Jul 01 17:16:39.129479 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.129419    2556 scope.go:117] "RemoveContainer" containerID="ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420"
Jul 01 17:16:39.131464 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.131446    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"]
Jul 01 17:16:39.136193 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.136173    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-df9a9-predictor-659d8d4f64-l4d2l"]
Jul 01 17:16:39.136808 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.136794    2556 scope.go:117] "RemoveContainer" containerID="cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f"
Jul 01 17:16:39.137083 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:16:39.137052    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f\": container with ID starting with cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f not found: ID does not exist" containerID="cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f"
Jul 01 17:16:39.137123 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.137093    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f"} err="failed to get container status \"cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f\": rpc error: code = NotFound desc = could not find container \"cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f\": container with ID starting with cadaf1bfb7de4877f953aaf5ce9ebe32d2828bff466ff66ca5893728b6fe979f not found: ID does not exist"
Jul 01 17:16:39.137123 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.137110    2556 scope.go:117] "RemoveContainer" containerID="15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c"
Jul 01 17:16:39.137343 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:16:39.137326    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c\": container with ID starting with 15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c not found: ID does not exist" containerID="15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c"
Jul 01 17:16:39.137379 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.137355    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c"} err="failed to get container status \"15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c\": rpc error: code = NotFound desc = could not find container \"15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c\": container with ID starting with 15ca4642c4af95c8d55ca5f76f29afdb7ca77e92d2427aa6583500823c10699c not found: ID does not exist"
Jul 01 17:16:39.137379 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.137373    2556 scope.go:117] "RemoveContainer" containerID="c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c"
Jul 01 17:16:39.137594 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:16:39.137578    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c\": container with ID starting with c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c not found: ID does not exist" containerID="c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c"
Jul 01 17:16:39.137643 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.137598    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c"} err="failed to get container status \"c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c\": rpc error: code = NotFound desc = could not find container \"c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c\": container with ID starting with c99a110994579a8469cb46ae4140bb8713df231f729f1c64fc627f0f33bd558c not found: ID does not exist"
Jul 01 17:16:39.137643 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.137611    2556 scope.go:117] "RemoveContainer" containerID="ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420"
Jul 01 17:16:39.137804 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:16:39.137788    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420\": container with ID starting with ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420 not found: ID does not exist" containerID="ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420"
Jul 01 17:16:39.137847 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.137808    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420"} err="failed to get container status \"ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420\": rpc error: code = NotFound desc = could not find container \"ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420\": container with ID starting with ce84dd3e1778fba857ea6f4a3731be5bbde19e18b1bd3b34fe3663d5c0090420 not found: ID does not exist"
Jul 01 17:16:39.195329 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.195299    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:39.195329 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.195326    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd62387-8aad-469a-ac5e-817e0e169f8b-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:39.195493 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.195339    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7mwb\" (UniqueName: \"kubernetes.io/projected/5cd62387-8aad-469a-ac5e-817e0e169f8b-kube-api-access-k7mwb\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:39.195493 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.195347    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd62387-8aad-469a-ac5e-817e0e169f8b-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:39.195493 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:39.195356    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd62387-8aad-469a-ac5e-817e0e169f8b-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:16:40.076388 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:40.076357    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" path="/var/lib/kubelet/pods/5cd62387-8aad-469a-ac5e-817e0e169f8b/volumes"
Jul 01 17:16:42.032036 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:42.031992    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:16:52.032110 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:16:52.032054    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:17:02.031349 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:17:02.031307    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:17:12.032212 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:17:12.032171    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:17:22.032081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:17:22.032019    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:17:32.031711 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:17:32.031669    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:17:35.068347 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:17:35.068302    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:17:45.069050 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:17:45.068999    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:17:55.068246 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:17:55.068195    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:18:05.068701 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:05.068660    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:18:15.068493 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:15.068447    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:18:25.068405 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:25.068359    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:18:30.634704 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:30.634673    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:18:30.637134 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:30.637109    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:18:35.068533 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:35.068495    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:18:45.068228 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:45.068185    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:18:55.068790 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:55.068756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:18:58.920511 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:58.920471    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"]
Jul 01 17:18:58.920966 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:58.920757    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" containerID="cri-o://3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84" gracePeriod=30
Jul 01 17:18:58.920966 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:58.920832    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kube-rbac-proxy" containerID="cri-o://6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276" gracePeriod=30
Jul 01 17:18:59.608285 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:59.608251    2556 generic.go:356] "Generic (PLEG): container finished" podID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerID="6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276" exitCode=2
Jul 01 17:18:59.608504 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:18:59.608328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerDied","Data":"6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276"}
Jul 01 17:19:02.026627 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:02.026581    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 01 17:19:04.049408 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049373    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"]
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049642    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049653    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049669    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kube-rbac-proxy"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049676    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kube-rbac-proxy"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049684    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="storage-initializer"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049690    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="storage-initializer"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049697    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049703    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049714    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kserve-container"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049719    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kserve-container"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049728    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container"
Jul 01 17:19:04.049809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049734    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container"
Jul 01 17:19:04.050224 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049822    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kube-rbac-proxy"
Jul 01 17:19:04.050224 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049830    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="agent"
Jul 01 17:19:04.050224 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049837    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kserve-container"
Jul 01 17:19:04.050224 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049843    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd62387-8aad-469a-ac5e-817e0e169f8b" containerName="kserve-container"
Jul 01 17:19:04.050224 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.049848    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b1cb1a-3439-48a1-90a7-245f435e8c54" containerName="kube-rbac-proxy"
Jul 01 17:19:04.052893 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.052876    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.055483 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.055463    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a14ffe-kube-rbac-proxy-sar-config\""
Jul 01 17:19:04.055590 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.055462    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a14ffe-predictor-serving-cert\""
Jul 01 17:19:04.065459 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.065435    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"]
Jul 01 17:19:04.083647 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.083618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.083795 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.083657    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cfb23dc-0276-417d-86b5-d764a69e07af-kserve-provision-location\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.083795 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.083682    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-cabundle-cert\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.083795 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.083768    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwcxv\" (UniqueName: \"kubernetes.io/projected/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-api-access-hwcxv\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.083950 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.083812    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.184752 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.184714    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwcxv\" (UniqueName: \"kubernetes.io/projected/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-api-access-hwcxv\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.184752 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.184760    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.184977 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.184815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.184977 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.184857    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cfb23dc-0276-417d-86b5-d764a69e07af-kserve-provision-location\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.184977 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.184892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-cabundle-cert\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.185171 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:19:04.185010    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-serving-cert: secret "isvc-primary-a14ffe-predictor-serving-cert" not found
Jul 01 17:19:04.185171 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:19:04.185103    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls podName:2cfb23dc-0276-417d-86b5-d764a69e07af nodeName:}" failed. No retries permitted until 2026-07-01 17:19:04.685056142 +0000 UTC m=+1235.027396807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls") pod "isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" (UID: "2cfb23dc-0276-417d-86b5-d764a69e07af") : secret "isvc-primary-a14ffe-predictor-serving-cert" not found
Jul 01 17:19:04.185358 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.185338    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cfb23dc-0276-417d-86b5-d764a69e07af-kserve-provision-location\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.185484 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.185467    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.185521 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.185500    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-cabundle-cert\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.194102 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.194081    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwcxv\" (UniqueName: \"kubernetes.io/projected/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-api-access-hwcxv\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.687887 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.687850    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.690296 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.690264    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls\") pod \"isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") " pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:04.962976 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:04.962866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:05.068558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:05.068518    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 01 17:19:05.097279 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:05.097250    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"]
Jul 01 17:19:05.100686 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:19:05.100658    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfb23dc_0276_417d_86b5_d764a69e07af.slice/crio-dc35b776c1edd460c70b1a68f6a2ef1af146764effac71585d953119bc59da59 WatchSource:0}: Error finding container dc35b776c1edd460c70b1a68f6a2ef1af146764effac71585d953119bc59da59: Status 404 returned error can't find the container with id dc35b776c1edd460c70b1a68f6a2ef1af146764effac71585d953119bc59da59
Jul 01 17:19:05.629638 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:05.629602    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerStarted","Data":"1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac"}
Jul 01 17:19:05.629844 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:05.629692    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerStarted","Data":"dc35b776c1edd460c70b1a68f6a2ef1af146764effac71585d953119bc59da59"}
Jul 01 17:19:07.026823 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:07.026782    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 01 17:19:08.914003 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.913981    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:19:08.918093 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918055    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-rbac-proxy-sar-config\") pod \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") "
Jul 01 17:19:08.918424 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918118    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-cabundle-cert\") pod \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") "
Jul 01 17:19:08.918424 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918142    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kserve-provision-location\") pod \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") "
Jul 01 17:19:08.918424 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918166    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2gtg\" (UniqueName: \"kubernetes.io/projected/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-api-access-h2gtg\") pod \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") "
Jul 01 17:19:08.918424 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918195    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls\") pod \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\" (UID: \"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9\") "
Jul 01 17:19:08.918619 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918433    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" (UID: "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:19:08.918619 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918465    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" (UID: "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:19:08.918619 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.918488    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" (UID: "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:19:08.920127 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.920101    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-api-access-h2gtg" (OuterVolumeSpecName: "kube-api-access-h2gtg") pod "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" (UID: "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9"). InnerVolumeSpecName "kube-api-access-h2gtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:19:08.920352 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:08.920329    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" (UID: "9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:19:09.019133 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.019092    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gtg\" (UniqueName: \"kubernetes.io/projected/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-api-access-h2gtg\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:19:09.019133 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.019122    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:19:09.019133 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.019132    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:19:09.019133 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.019141    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:19:09.019398 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.019150    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:19:09.644635 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.644603    2556 generic.go:356] "Generic (PLEG): container finished" podID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerID="1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac" exitCode=0
Jul 01 17:19:09.644816 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.644674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerDied","Data":"1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac"}
Jul 01 17:19:09.646556 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.646533    2556 generic.go:356] "Generic (PLEG): container finished" podID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerID="3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84" exitCode=0
Jul 01 17:19:09.646677 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.646593    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"
Jul 01 17:19:09.646677 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.646614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerDied","Data":"3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84"}
Jul 01 17:19:09.646677 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.646664    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd" event={"ID":"9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9","Type":"ContainerDied","Data":"fad0926a930f966508812fb413850028eefc1a5d4db81356b365c84aa2dee971"}
Jul 01 17:19:09.646832 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.646693    2556 scope.go:117] "RemoveContainer" containerID="6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276"
Jul 01 17:19:09.657851 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.657833    2556 scope.go:117] "RemoveContainer" containerID="3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84"
Jul 01 17:19:09.670912 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.670896    2556 scope.go:117] "RemoveContainer" containerID="db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343"
Jul 01 17:19:09.680474 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.680453    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"]
Jul 01 17:19:09.683521 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.683504    2556 scope.go:117] "RemoveContainer" containerID="6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276"
Jul 01 17:19:09.683772 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:19:09.683753    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276\": container with ID starting with 6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276 not found: ID does not exist" containerID="6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276"
Jul 01 17:19:09.683835 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.683781    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276"} err="failed to get container status \"6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276\": rpc error: code = NotFound desc = could not find container \"6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276\": container with ID starting with 6df08d792d741fd7c2e4d8cd974f7cdd75db6a5700469c4aa5f24fb461ab0276 not found: ID does not exist"
Jul 01 17:19:09.683835 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.683804    2556 scope.go:117] "RemoveContainer" containerID="3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84"
Jul 01 17:19:09.684379 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:19:09.684357    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84\": container with ID starting with 3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84 not found: ID does not exist" containerID="3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84"
Jul 01 17:19:09.684460 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.684386    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84"} err="failed to get container status \"3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84\": rpc error: code = NotFound desc = could not find container \"3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84\": container with ID starting with 3fcee97fe402e872e67461e9913b9baa3b813c2069633ef15ab2280ae0e40e84 not found: ID does not exist"
Jul 01 17:19:09.684460 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.684411    2556 scope.go:117] "RemoveContainer" containerID="db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343"
Jul 01 17:19:09.684807 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:19:09.684745    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343\": container with ID starting with db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343 not found: ID does not exist" containerID="db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343"
Jul 01 17:19:09.684807 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.684783    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343"} err="failed to get container status \"db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343\": rpc error: code = NotFound desc = could not find container \"db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343\": container with ID starting with db6210b1c32d52388f3c6683130b8c061367532d23222f65f9739159cab4b343 not found: ID does not exist"
Jul 01 17:19:09.686533 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:09.686515    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-76ae4-predictor-6b8f655f68-vknpd"]
Jul 01 17:19:10.073541 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:10.073505    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" path="/var/lib/kubelet/pods/9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9/volumes"
Jul 01 17:19:10.652404 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:10.652369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerStarted","Data":"eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c"}
Jul 01 17:19:10.652579 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:10.652436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerStarted","Data":"dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a"}
Jul 01 17:19:10.652799 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:10.652778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:10.652930 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:10.652892    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:10.654522 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:10.654494    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:19:10.674939 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:10.674897    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podStartSLOduration=2.992174743 podStartE2EDuration="6.674869466s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.682694723s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:19:10.673756637 +0000 UTC m=+1241.016097299" watchObservedRunningTime="2026-07-01 17:19:10.674869466 +0000 UTC m=+1241.017210125"
Jul 01 17:19:11.657664 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:11.657624    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:19:16.662714 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:16.662684    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:19:16.663202 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:16.663178    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:19:26.663937 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:26.663851    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:19:36.663529 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:36.663483    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:19:46.663274 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:46.663227    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:19:56.664170 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:19:56.664132    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:20:06.663506 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:06.663466    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:20:16.663505 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:16.663465    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 01 17:20:26.664236 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:26.664202    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:20:34.286429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286397    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"]
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286652    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kube-rbac-proxy"
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286662    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kube-rbac-proxy"
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286674    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container"
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286680    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container"
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286706    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="storage-initializer"
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286712    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="storage-initializer"
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286759    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kube-rbac-proxy"
Jul 01 17:20:34.286884 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.286766    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f04d4e1-4be5-45b9-a6d3-e69a2274cdf9" containerName="kserve-container"
Jul 01 17:20:34.289832 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.289815    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.293264 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.293243    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a14ffe-predictor-serving-cert\""
Jul 01 17:20:34.293555 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.293536    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a14ffe-kube-rbac-proxy-sar-config\""
Jul 01 17:20:34.293635 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.293539    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a14ffe\""
Jul 01 17:20:34.294013 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.294000    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a14ffe-dockercfg-l865p\""
Jul 01 17:20:34.302943 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.302921    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"]
Jul 01 17:20:34.371749 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.371716    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-cabundle-cert\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.371917 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.371758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bq2n\" (UniqueName: \"kubernetes.io/projected/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-api-access-6bq2n\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.371917 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.371778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.371917 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.371809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12a9898-14b9-4c78-b2ed-feaa38530b96-kserve-provision-location\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.371917 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.371845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b12a9898-14b9-4c78-b2ed-feaa38530b96-proxy-tls\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.472994 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.472959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-cabundle-cert\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.473204 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.473003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bq2n\" (UniqueName: \"kubernetes.io/projected/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-api-access-6bq2n\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.473204 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.473025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.473204 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.473043    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12a9898-14b9-4c78-b2ed-feaa38530b96-kserve-provision-location\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.473204 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.473090    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b12a9898-14b9-4c78-b2ed-feaa38530b96-proxy-tls\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.473537 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.473521    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12a9898-14b9-4c78-b2ed-feaa38530b96-kserve-provision-location\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.473786 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.473768    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-cabundle-cert\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.473830 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.473780    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.475616 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.475599    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b12a9898-14b9-4c78-b2ed-feaa38530b96-proxy-tls\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.483919 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.483899    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bq2n\" (UniqueName: \"kubernetes.io/projected/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-api-access-6bq2n\") pod \"isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") " pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.599438 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.599340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:34.729342 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.729314    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"]
Jul 01 17:20:34.731022 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:20:34.730999    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12a9898_14b9_4c78_b2ed_feaa38530b96.slice/crio-200bd79e25f5e744c1211345abea06d636a4e86b61edd09d667d0d49a5aa5a62 WatchSource:0}: Error finding container 200bd79e25f5e744c1211345abea06d636a4e86b61edd09d667d0d49a5aa5a62: Status 404 returned error can't find the container with id 200bd79e25f5e744c1211345abea06d636a4e86b61edd09d667d0d49a5aa5a62
Jul 01 17:20:34.949334 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.949239    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q" event={"ID":"b12a9898-14b9-4c78-b2ed-feaa38530b96","Type":"ContainerStarted","Data":"0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781"}
Jul 01 17:20:34.949334 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:34.949318    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q" event={"ID":"b12a9898-14b9-4c78-b2ed-feaa38530b96","Type":"ContainerStarted","Data":"200bd79e25f5e744c1211345abea06d636a4e86b61edd09d667d0d49a5aa5a62"}
Jul 01 17:20:37.961834 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:37.961805    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_b12a9898-14b9-4c78-b2ed-feaa38530b96/storage-initializer/0.log"
Jul 01 17:20:37.962213 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:37.961842    2556 generic.go:356] "Generic (PLEG): container finished" podID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerID="0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781" exitCode=1
Jul 01 17:20:37.962213 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:37.961895    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q" event={"ID":"b12a9898-14b9-4c78-b2ed-feaa38530b96","Type":"ContainerDied","Data":"0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781"}
Jul 01 17:20:38.967007 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:38.966979    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_b12a9898-14b9-4c78-b2ed-feaa38530b96/storage-initializer/0.log"
Jul 01 17:20:38.967517 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:38.967102    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q" event={"ID":"b12a9898-14b9-4c78-b2ed-feaa38530b96","Type":"ContainerStarted","Data":"56a42def8783fd1222d17cd8decd5b990ae061304030108503ffb26ef28bbb52"}
Jul 01 17:20:41.980007 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:41.979977    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_b12a9898-14b9-4c78-b2ed-feaa38530b96/storage-initializer/1.log"
Jul 01 17:20:41.980506 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:41.980359    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_b12a9898-14b9-4c78-b2ed-feaa38530b96/storage-initializer/0.log"
Jul 01 17:20:41.980506 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:41.980400    2556 generic.go:356] "Generic (PLEG): container finished" podID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerID="56a42def8783fd1222d17cd8decd5b990ae061304030108503ffb26ef28bbb52" exitCode=1
Jul 01 17:20:41.980506 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:41.980447    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q" event={"ID":"b12a9898-14b9-4c78-b2ed-feaa38530b96","Type":"ContainerDied","Data":"56a42def8783fd1222d17cd8decd5b990ae061304030108503ffb26ef28bbb52"}
Jul 01 17:20:41.980666 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:41.980552    2556 scope.go:117] "RemoveContainer" containerID="0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781"
Jul 01 17:20:41.980938 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:41.980922    2556 scope.go:117] "RemoveContainer" containerID="0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781"
Jul 01 17:20:41.990941 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:20:41.990914    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_kserve-ci-e2e-test(b12a9898-14b9-4c78-b2ed-feaa38530b96)\"" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96"
Jul 01 17:20:41.991011 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:20:41.990972    2556 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_kserve-ci-e2e-test_b12a9898-14b9-4c78-b2ed-feaa38530b96_0 in pod sandbox 200bd79e25f5e744c1211345abea06d636a4e86b61edd09d667d0d49a5aa5a62 from index: no such id: '0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781'" containerID="0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781"
Jul 01 17:20:41.991086 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:41.991006    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_kserve-ci-e2e-test_b12a9898-14b9-4c78-b2ed-feaa38530b96_0 in pod sandbox 200bd79e25f5e744c1211345abea06d636a4e86b61edd09d667d0d49a5aa5a62 from index: no such id: '0adb7d423434f9614a17075cdb638210c1e1f2cafa695797328a7474e3697781'"
Jul 01 17:20:42.987422 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:42.987395    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_b12a9898-14b9-4c78-b2ed-feaa38530b96/storage-initializer/1.log"
Jul 01 17:20:48.291702 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.291661    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"]
Jul 01 17:20:48.292168 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.291966    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container" containerID="cri-o://dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a" gracePeriod=30
Jul 01 17:20:48.292168 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.292029    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kube-rbac-proxy" containerID="cri-o://eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c" gracePeriod=30
Jul 01 17:20:48.350256 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.350222    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"]
Jul 01 17:20:48.488519 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.488498    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_b12a9898-14b9-4c78-b2ed-feaa38530b96/storage-initializer/1.log"
Jul 01 17:20:48.490779 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.490765    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:48.558767 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.558686    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bq2n\" (UniqueName: \"kubernetes.io/projected/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-api-access-6bq2n\") pod \"b12a9898-14b9-4c78-b2ed-feaa38530b96\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") "
Jul 01 17:20:48.558767 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.558736    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12a9898-14b9-4c78-b2ed-feaa38530b96-kserve-provision-location\") pod \"b12a9898-14b9-4c78-b2ed-feaa38530b96\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") "
Jul 01 17:20:48.558945 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.558776    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b12a9898-14b9-4c78-b2ed-feaa38530b96-proxy-tls\") pod \"b12a9898-14b9-4c78-b2ed-feaa38530b96\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") "
Jul 01 17:20:48.558945 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.558836    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-cabundle-cert\") pod \"b12a9898-14b9-4c78-b2ed-feaa38530b96\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") "
Jul 01 17:20:48.558945 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.558868    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-rbac-proxy-sar-config\") pod \"b12a9898-14b9-4c78-b2ed-feaa38530b96\" (UID: \"b12a9898-14b9-4c78-b2ed-feaa38530b96\") "
Jul 01 17:20:48.559091 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.559033    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a9898-14b9-4c78-b2ed-feaa38530b96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b12a9898-14b9-4c78-b2ed-feaa38530b96" (UID: "b12a9898-14b9-4c78-b2ed-feaa38530b96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:20:48.559282 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.559255    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b12a9898-14b9-4c78-b2ed-feaa38530b96" (UID: "b12a9898-14b9-4c78-b2ed-feaa38530b96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:48.559415 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.559289    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b12a9898-14b9-4c78-b2ed-feaa38530b96" (UID: "b12a9898-14b9-4c78-b2ed-feaa38530b96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:48.560899 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.560872    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-api-access-6bq2n" (OuterVolumeSpecName: "kube-api-access-6bq2n") pod "b12a9898-14b9-4c78-b2ed-feaa38530b96" (UID: "b12a9898-14b9-4c78-b2ed-feaa38530b96"). InnerVolumeSpecName "kube-api-access-6bq2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:48.561015 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.560917    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12a9898-14b9-4c78-b2ed-feaa38530b96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b12a9898-14b9-4c78-b2ed-feaa38530b96" (UID: "b12a9898-14b9-4c78-b2ed-feaa38530b96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:48.660160 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.660122    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b12a9898-14b9-4c78-b2ed-feaa38530b96-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:48.660160 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.660152    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b12a9898-14b9-4c78-b2ed-feaa38530b96-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:48.660160 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.660161    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:48.660390 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.660170    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:48.660390 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:48.660181    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bq2n\" (UniqueName: \"kubernetes.io/projected/b12a9898-14b9-4c78-b2ed-feaa38530b96-kube-api-access-6bq2n\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:49.011022 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.010992    2556 generic.go:356] "Generic (PLEG): container finished" podID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerID="eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c" exitCode=2
Jul 01 17:20:49.011203 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.011087    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerDied","Data":"eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c"}
Jul 01 17:20:49.012272 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.012253    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q_b12a9898-14b9-4c78-b2ed-feaa38530b96/storage-initializer/1.log"
Jul 01 17:20:49.012402 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.012309    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q" event={"ID":"b12a9898-14b9-4c78-b2ed-feaa38530b96","Type":"ContainerDied","Data":"200bd79e25f5e744c1211345abea06d636a4e86b61edd09d667d0d49a5aa5a62"}
Jul 01 17:20:49.012402 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.012341    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"
Jul 01 17:20:49.012402 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.012370    2556 scope.go:117] "RemoveContainer" containerID="56a42def8783fd1222d17cd8decd5b990ae061304030108503ffb26ef28bbb52"
Jul 01 17:20:49.054903 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.054870    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"]
Jul 01 17:20:49.057957 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:49.057930    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a14ffe-predictor-6dbcf6bb68-5gh8q"]
Jul 01 17:20:50.073445 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:50.073413    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96" path="/var/lib/kubelet/pods/b12a9898-14b9-4c78-b2ed-feaa38530b96/volumes"
Jul 01 17:20:51.658497 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:51.658407    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 01 17:20:52.941579 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.941556    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:20:52.991581 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991552    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-cabundle-cert\") pod \"2cfb23dc-0276-417d-86b5-d764a69e07af\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") "
Jul 01 17:20:52.991734 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991588    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cfb23dc-0276-417d-86b5-d764a69e07af-kserve-provision-location\") pod \"2cfb23dc-0276-417d-86b5-d764a69e07af\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") "
Jul 01 17:20:52.991734 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991617    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-rbac-proxy-sar-config\") pod \"2cfb23dc-0276-417d-86b5-d764a69e07af\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") "
Jul 01 17:20:52.991734 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991652    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls\") pod \"2cfb23dc-0276-417d-86b5-d764a69e07af\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") "
Jul 01 17:20:52.991941 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991761    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwcxv\" (UniqueName: \"kubernetes.io/projected/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-api-access-hwcxv\") pod \"2cfb23dc-0276-417d-86b5-d764a69e07af\" (UID: \"2cfb23dc-0276-417d-86b5-d764a69e07af\") "
Jul 01 17:20:52.992004 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991952    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cfb23dc-0276-417d-86b5-d764a69e07af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2cfb23dc-0276-417d-86b5-d764a69e07af" (UID: "2cfb23dc-0276-417d-86b5-d764a69e07af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:20:52.992004 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991970    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2cfb23dc-0276-417d-86b5-d764a69e07af" (UID: "2cfb23dc-0276-417d-86b5-d764a69e07af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:52.992004 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.991988    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2cfb23dc-0276-417d-86b5-d764a69e07af" (UID: "2cfb23dc-0276-417d-86b5-d764a69e07af"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:20:52.993764 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.993730    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-api-access-hwcxv" (OuterVolumeSpecName: "kube-api-access-hwcxv") pod "2cfb23dc-0276-417d-86b5-d764a69e07af" (UID: "2cfb23dc-0276-417d-86b5-d764a69e07af"). InnerVolumeSpecName "kube-api-access-hwcxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:20:52.993868 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:52.993824    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2cfb23dc-0276-417d-86b5-d764a69e07af" (UID: "2cfb23dc-0276-417d-86b5-d764a69e07af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:20:53.028714 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.028652    2556 generic.go:356] "Generic (PLEG): container finished" podID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerID="dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a" exitCode=0
Jul 01 17:20:53.028714 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.028709    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"
Jul 01 17:20:53.028886 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.028726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerDied","Data":"dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a"}
Jul 01 17:20:53.028886 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.028818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb" event={"ID":"2cfb23dc-0276-417d-86b5-d764a69e07af","Type":"ContainerDied","Data":"dc35b776c1edd460c70b1a68f6a2ef1af146764effac71585d953119bc59da59"}
Jul 01 17:20:53.028886 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.028858    2556 scope.go:117] "RemoveContainer" containerID="eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c"
Jul 01 17:20:53.036558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.036537    2556 scope.go:117] "RemoveContainer" containerID="dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a"
Jul 01 17:20:53.043242 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.043228    2556 scope.go:117] "RemoveContainer" containerID="1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac"
Jul 01 17:20:53.049830 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.049811    2556 scope.go:117] "RemoveContainer" containerID="eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c"
Jul 01 17:20:53.050083 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:20:53.050047    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c\": container with ID starting with eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c not found: ID does not exist" containerID="eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c"
Jul 01 17:20:53.050162 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.050093    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c"} err="failed to get container status \"eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c\": rpc error: code = NotFound desc = could not find container \"eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c\": container with ID starting with eb5dcb62ce039b1b202552e59fabdb3cd9c47553b1384f97066defc76c4d013c not found: ID does not exist"
Jul 01 17:20:53.050162 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.050116    2556 scope.go:117] "RemoveContainer" containerID="dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a"
Jul 01 17:20:53.050343 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:20:53.050327    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a\": container with ID starting with dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a not found: ID does not exist" containerID="dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a"
Jul 01 17:20:53.050386 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.050349    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a"} err="failed to get container status \"dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a\": rpc error: code = NotFound desc = could not find container \"dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a\": container with ID starting with dc0140bd18bda468bb61a30eea97e7a2e616000ea53d9e231c0911afdf2e458a not found: ID does not exist"
Jul 01 17:20:53.050386 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.050364    2556 scope.go:117] "RemoveContainer" containerID="1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac"
Jul 01 17:20:53.050550 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:20:53.050529    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac\": container with ID starting with 1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac not found: ID does not exist" containerID="1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac"
Jul 01 17:20:53.050588 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.050560    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac"} err="failed to get container status \"1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac\": rpc error: code = NotFound desc = could not find container \"1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac\": container with ID starting with 1390d0c6acba6a877e46209ac993c080cf4538018ab4472f81a4866d0beacdac not found: ID does not exist"
Jul 01 17:20:53.056307 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.056287    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"]
Jul 01 17:20:53.065302 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.065278    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a14ffe-predictor-7b957ccb6c-j9lvb"]
Jul 01 17:20:53.093081 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.093042    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:53.093176 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.093087    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cfb23dc-0276-417d-86b5-d764a69e07af-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:53.093176 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.093103    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwcxv\" (UniqueName: \"kubernetes.io/projected/2cfb23dc-0276-417d-86b5-d764a69e07af-kube-api-access-hwcxv\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:53.093176 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.093116    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cfb23dc-0276-417d-86b5-d764a69e07af-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:53.093176 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.093130    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cfb23dc-0276-417d-86b5-d764a69e07af-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:20:53.499301 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499264    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"]
Jul 01 17:20:53.499539 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499525    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="storage-initializer"
Jul 01 17:20:53.499582 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499542    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="storage-initializer"
Jul 01 17:20:53.499582 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499557    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerName="storage-initializer"
Jul 01 17:20:53.499582 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499564    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerName="storage-initializer"
Jul 01 17:20:53.499582 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499575    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container"
Jul 01 17:20:53.499582 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499581    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499589    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kube-rbac-proxy"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499596    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kube-rbac-proxy"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499606    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerName="storage-initializer"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499611    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerName="storage-initializer"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499661    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kube-rbac-proxy"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499668    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerName="storage-initializer"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499674    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12a9898-14b9-4c78-b2ed-feaa38530b96" containerName="storage-initializer"
Jul 01 17:20:53.499736 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.499680    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" containerName="kserve-container"
Jul 01 17:20:53.503928 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.503910    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.506775 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.506756    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-9c3795-dockercfg-97hcz\""
Jul 01 17:20:53.506775 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.506756    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:20:53.506931 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.506780    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-9c3795\""
Jul 01 17:20:53.506931 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.506777    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9c3795-kube-rbac-proxy-sar-config\""
Jul 01 17:20:53.507117 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.507104    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9c3795-predictor-serving-cert\""
Jul 01 17:20:53.507860 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.507841    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:20:53.507945 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.507844    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:20:53.518303 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.518281    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"]
Jul 01 17:20:53.596967 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.596930    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-cabundle-cert\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.597141 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.596979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.597141 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.597012    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.597141 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.597090    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4cl\" (UniqueName: \"kubernetes.io/projected/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-api-access-8j4cl\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.597141 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.597123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af617931-4ed8-4989-8b3e-ac8006fb9e32-kserve-provision-location\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.698442 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.698406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-cabundle-cert\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.698548 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.698461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.698548 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.698489    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.698548 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.698511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4cl\" (UniqueName: \"kubernetes.io/projected/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-api-access-8j4cl\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.698548 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.698531    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af617931-4ed8-4989-8b3e-ac8006fb9e32-kserve-provision-location\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.698717 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:20:53.698656    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-serving-cert: secret "isvc-init-fail-9c3795-predictor-serving-cert" not found
Jul 01 17:20:53.698755 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:20:53.698740    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls podName:af617931-4ed8-4989-8b3e-ac8006fb9e32 nodeName:}" failed. No retries permitted until 2026-07-01 17:20:54.198717997 +0000 UTC m=+1344.541058645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls") pod "isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" (UID: "af617931-4ed8-4989-8b3e-ac8006fb9e32") : secret "isvc-init-fail-9c3795-predictor-serving-cert" not found
Jul 01 17:20:53.699011 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.698990    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af617931-4ed8-4989-8b3e-ac8006fb9e32-kserve-provision-location\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.699312 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.699291    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.699356 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.699291    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-cabundle-cert\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:53.707923 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:53.707899    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4cl\" (UniqueName: \"kubernetes.io/projected/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-api-access-8j4cl\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:54.072662 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:54.072627    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cfb23dc-0276-417d-86b5-d764a69e07af" path="/var/lib/kubelet/pods/2cfb23dc-0276-417d-86b5-d764a69e07af/volumes"
Jul 01 17:20:54.202377 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:54.202337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:54.204744 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:54.204718    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls\") pod \"isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:54.413837 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:54.413743    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:20:54.542346 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:54.542315    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"]
Jul 01 17:20:54.544736 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:20:54.544706    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf617931_4ed8_4989_8b3e_ac8006fb9e32.slice/crio-fa5252804c3e80a465fe323b00370cc63781f6775b43b318ff9f5c0844548195 WatchSource:0}: Error finding container fa5252804c3e80a465fe323b00370cc63781f6775b43b318ff9f5c0844548195: Status 404 returned error can't find the container with id fa5252804c3e80a465fe323b00370cc63781f6775b43b318ff9f5c0844548195
Jul 01 17:20:55.037864 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:55.037826    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" event={"ID":"af617931-4ed8-4989-8b3e-ac8006fb9e32","Type":"ContainerStarted","Data":"dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba"}
Jul 01 17:20:55.038115 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:20:55.037892    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" event={"ID":"af617931-4ed8-4989-8b3e-ac8006fb9e32","Type":"ContainerStarted","Data":"fa5252804c3e80a465fe323b00370cc63781f6775b43b318ff9f5c0844548195"}
Jul 01 17:21:00.057714 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:00.057679    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2_af617931-4ed8-4989-8b3e-ac8006fb9e32/storage-initializer/0.log"
Jul 01 17:21:00.058154 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:00.057720    2556 generic.go:356] "Generic (PLEG): container finished" podID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerID="dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba" exitCode=1
Jul 01 17:21:00.058154 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:00.057796    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" event={"ID":"af617931-4ed8-4989-8b3e-ac8006fb9e32","Type":"ContainerDied","Data":"dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba"}
Jul 01 17:21:01.063768 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:01.063741    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2_af617931-4ed8-4989-8b3e-ac8006fb9e32/storage-initializer/0.log"
Jul 01 17:21:01.064167 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:01.063794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" event={"ID":"af617931-4ed8-4989-8b3e-ac8006fb9e32","Type":"ContainerStarted","Data":"6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225"}
Jul 01 17:21:03.502576 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:03.502544    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"]
Jul 01 17:21:03.503040 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:03.502840    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerName="storage-initializer" containerID="cri-o://6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225" gracePeriod=30
Jul 01 17:21:05.451422 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.451393    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2_af617931-4ed8-4989-8b3e-ac8006fb9e32/storage-initializer/1.log"
Jul 01 17:21:05.451858 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.451841    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2_af617931-4ed8-4989-8b3e-ac8006fb9e32/storage-initializer/0.log"
Jul 01 17:21:05.454170 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.454146    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:21:05.485645 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.485612    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af617931-4ed8-4989-8b3e-ac8006fb9e32-kserve-provision-location\") pod \"af617931-4ed8-4989-8b3e-ac8006fb9e32\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") "
Jul 01 17:21:05.485780 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.485663    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j4cl\" (UniqueName: \"kubernetes.io/projected/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-api-access-8j4cl\") pod \"af617931-4ed8-4989-8b3e-ac8006fb9e32\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") "
Jul 01 17:21:05.485780 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.485698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-cabundle-cert\") pod \"af617931-4ed8-4989-8b3e-ac8006fb9e32\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") "
Jul 01 17:21:05.485780 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.485741    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls\") pod \"af617931-4ed8-4989-8b3e-ac8006fb9e32\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") "
Jul 01 17:21:05.485908 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.485886    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af617931-4ed8-4989-8b3e-ac8006fb9e32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "af617931-4ed8-4989-8b3e-ac8006fb9e32" (UID: "af617931-4ed8-4989-8b3e-ac8006fb9e32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:21:05.485955 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.485900    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-rbac-proxy-sar-config\") pod \"af617931-4ed8-4989-8b3e-ac8006fb9e32\" (UID: \"af617931-4ed8-4989-8b3e-ac8006fb9e32\") "
Jul 01 17:21:05.486147 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.486116    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/af617931-4ed8-4989-8b3e-ac8006fb9e32-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:21:05.486216 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.486140    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "af617931-4ed8-4989-8b3e-ac8006fb9e32" (UID: "af617931-4ed8-4989-8b3e-ac8006fb9e32"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:21:05.486257 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.486211    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "af617931-4ed8-4989-8b3e-ac8006fb9e32" (UID: "af617931-4ed8-4989-8b3e-ac8006fb9e32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:21:05.487855 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.487831    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af617931-4ed8-4989-8b3e-ac8006fb9e32" (UID: "af617931-4ed8-4989-8b3e-ac8006fb9e32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:21:05.487944 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.487907    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-api-access-8j4cl" (OuterVolumeSpecName: "kube-api-access-8j4cl") pod "af617931-4ed8-4989-8b3e-ac8006fb9e32" (UID: "af617931-4ed8-4989-8b3e-ac8006fb9e32"). InnerVolumeSpecName "kube-api-access-8j4cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:21:05.586896 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.586857    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:21:05.586896 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.586887    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j4cl\" (UniqueName: \"kubernetes.io/projected/af617931-4ed8-4989-8b3e-ac8006fb9e32-kube-api-access-8j4cl\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:21:05.586896 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.586900    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/af617931-4ed8-4989-8b3e-ac8006fb9e32-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:21:05.587168 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:05.586913    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af617931-4ed8-4989-8b3e-ac8006fb9e32-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:21:06.082657 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.082635    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2_af617931-4ed8-4989-8b3e-ac8006fb9e32/storage-initializer/1.log"
Jul 01 17:21:06.082994 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.082980    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2_af617931-4ed8-4989-8b3e-ac8006fb9e32/storage-initializer/0.log"
Jul 01 17:21:06.083045 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.083016    2556 generic.go:356] "Generic (PLEG): container finished" podID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerID="6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225" exitCode=1
Jul 01 17:21:06.083104 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.083088    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"
Jul 01 17:21:06.083162 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.083095    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" event={"ID":"af617931-4ed8-4989-8b3e-ac8006fb9e32","Type":"ContainerDied","Data":"6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225"}
Jul 01 17:21:06.083219 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.083172    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2" event={"ID":"af617931-4ed8-4989-8b3e-ac8006fb9e32","Type":"ContainerDied","Data":"fa5252804c3e80a465fe323b00370cc63781f6775b43b318ff9f5c0844548195"}
Jul 01 17:21:06.083269 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.083216    2556 scope.go:117] "RemoveContainer" containerID="6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225"
Jul 01 17:21:06.090830 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.090816    2556 scope.go:117] "RemoveContainer" containerID="dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba"
Jul 01 17:21:06.097633 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.097607    2556 scope.go:117] "RemoveContainer" containerID="6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225"
Jul 01 17:21:06.097873 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:21:06.097846    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225\": container with ID starting with 6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225 not found: ID does not exist" containerID="6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225"
Jul 01 17:21:06.097930 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.097878    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225"} err="failed to get container status \"6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225\": rpc error: code = NotFound desc = could not find container \"6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225\": container with ID starting with 6e5d89b0965e19be078b4a45db7fff5ecd8c6e9b886a5c1415201046dc6c3225 not found: ID does not exist"
Jul 01 17:21:06.097930 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.097897    2556 scope.go:117] "RemoveContainer" containerID="dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba"
Jul 01 17:21:06.098173 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:21:06.098149    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba\": container with ID starting with dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba not found: ID does not exist" containerID="dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba"
Jul 01 17:21:06.098223 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.098171    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba"} err="failed to get container status \"dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba\": rpc error: code = NotFound desc = could not find container \"dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba\": container with ID starting with dc35b08f30d5d15e264199e4d7d8be973e871692066257aeac14c1449ad737ba not found: ID does not exist"
Jul 01 17:21:06.120687 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.120662    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"]
Jul 01 17:21:06.126266 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:06.126241    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c3795-predictor-7cb8c994db-4tkb2"]
Jul 01 17:21:08.072865 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.072822    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" path="/var/lib/kubelet/pods/af617931-4ed8-4989-8b3e-ac8006fb9e32/volumes"
Jul 01 17:21:08.724014 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.723975    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"]
Jul 01 17:21:08.724348 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.724330    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerName="storage-initializer"
Jul 01 17:21:08.724457 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.724351    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerName="storage-initializer"
Jul 01 17:21:08.724457 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.724374    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerName="storage-initializer"
Jul 01 17:21:08.724457 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.724383    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerName="storage-initializer"
Jul 01 17:21:08.724609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.724462    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerName="storage-initializer"
Jul 01 17:21:08.724609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.724474    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af617931-4ed8-4989-8b3e-ac8006fb9e32" containerName="storage-initializer"
Jul 01 17:21:08.729087 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.729043    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.732300 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.732279    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g2mrd\""
Jul 01 17:21:08.732399 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.732302    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-08781-predictor-serving-cert\""
Jul 01 17:21:08.732399 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.732281    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-08781-kube-rbac-proxy-sar-config\""
Jul 01 17:21:08.732910 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.732892    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:21:08.732910 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.732902    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:21:08.733323 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.733308    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:21:08.733454 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.733441    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:21:08.740956 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.740933    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"]
Jul 01 17:21:08.812872 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.812835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.813085 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.812877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-cabundle-cert\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.813085 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.812913    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5450776-16b9-4d97-8dfc-c77ce263c199-kserve-provision-location\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.813085 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.812949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgsp\" (UniqueName: \"kubernetes.io/projected/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-api-access-zkgsp\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.813085 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.813029    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.913813 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.913768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.913813 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.913819    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-cabundle-cert\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.914118 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.913849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5450776-16b9-4d97-8dfc-c77ce263c199-kserve-provision-location\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.914118 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.913886    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgsp\" (UniqueName: \"kubernetes.io/projected/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-api-access-zkgsp\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.914118 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.913926    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.914118 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:21:08.913954    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-08781-predictor-serving-cert: secret "raw-sklearn-08781-predictor-serving-cert" not found
Jul 01 17:21:08.914118 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:21:08.914020    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls podName:d5450776-16b9-4d97-8dfc-c77ce263c199 nodeName:}" failed. No retries permitted until 2026-07-01 17:21:09.414003212 +0000 UTC m=+1359.756343853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls") pod "raw-sklearn-08781-predictor-66bbc8c58-tszss" (UID: "d5450776-16b9-4d97-8dfc-c77ce263c199") : secret "raw-sklearn-08781-predictor-serving-cert" not found
Jul 01 17:21:08.914380 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.914335    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5450776-16b9-4d97-8dfc-c77ce263c199-kserve-provision-location\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.914600 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.914577    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-cabundle-cert\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.914740 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.914721    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:08.923577 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:08.923555    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgsp\" (UniqueName: \"kubernetes.io/projected/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-api-access-zkgsp\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:09.418172 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:09.418124    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:09.418544 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:21:09.418292    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-08781-predictor-serving-cert: secret "raw-sklearn-08781-predictor-serving-cert" not found
Jul 01 17:21:09.418544 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:21:09.418359    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls podName:d5450776-16b9-4d97-8dfc-c77ce263c199 nodeName:}" failed. No retries permitted until 2026-07-01 17:21:10.41834394 +0000 UTC m=+1360.760684578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls") pod "raw-sklearn-08781-predictor-66bbc8c58-tszss" (UID: "d5450776-16b9-4d97-8dfc-c77ce263c199") : secret "raw-sklearn-08781-predictor-serving-cert" not found
Jul 01 17:21:10.428287 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:10.428254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:10.430635 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:10.430615    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls\") pod \"raw-sklearn-08781-predictor-66bbc8c58-tszss\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") " pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:10.539188 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:10.539137    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:10.673689 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:10.673663    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"]
Jul 01 17:21:10.676138 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:21:10.676112    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5450776_16b9_4d97_8dfc_c77ce263c199.slice/crio-f601ef7d2f9f41e12d780ca1020a991ec1b6c962204a372fa19159c9021ffed6 WatchSource:0}: Error finding container f601ef7d2f9f41e12d780ca1020a991ec1b6c962204a372fa19159c9021ffed6: Status 404 returned error can't find the container with id f601ef7d2f9f41e12d780ca1020a991ec1b6c962204a372fa19159c9021ffed6
Jul 01 17:21:11.104111 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:11.104039    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerStarted","Data":"89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f"}
Jul 01 17:21:11.104261 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:11.104156    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerStarted","Data":"f601ef7d2f9f41e12d780ca1020a991ec1b6c962204a372fa19159c9021ffed6"}
Jul 01 17:21:15.120824 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:15.120794    2556 generic.go:356] "Generic (PLEG): container finished" podID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerID="89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f" exitCode=0
Jul 01 17:21:15.121286 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:15.120839    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerDied","Data":"89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f"}
Jul 01 17:21:16.126074 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:16.126022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerStarted","Data":"06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614"}
Jul 01 17:21:16.126454 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:16.126107    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerStarted","Data":"4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44"}
Jul 01 17:21:16.126454 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:16.126393    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:16.126533 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:16.126522    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:16.127799 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:16.127770    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:21:16.147493 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:16.147441    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podStartSLOduration=4.321335528 podStartE2EDuration="8.147425294s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.826089766s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:21:16.146547582 +0000 UTC m=+1366.488888275" watchObservedRunningTime="2026-07-01 17:21:16.147425294 +0000 UTC m=+1366.489765953"
Jul 01 17:21:17.130338 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:17.130297    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:21:22.134348 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:22.134317    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:21:22.134820 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:22.134794    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:21:32.134809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:32.134764    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:21:42.134838 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:42.134794    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:21:52.134771 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:21:52.134726    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:22:02.135452 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:02.135410    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:22:12.135551 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:12.135513    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:22:22.135346 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:22.135261    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:22:32.136155 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:32.136125    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:22:39.002803 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:39.002768    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"]
Jul 01 17:22:39.003205 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:39.003040    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" containerID="cri-o://4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44" gracePeriod=30
Jul 01 17:22:39.003205 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:39.003088    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kube-rbac-proxy" containerID="cri-o://06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614" gracePeriod=30
Jul 01 17:22:39.414747 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:39.414660    2556 generic.go:356] "Generic (PLEG): container finished" podID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerID="06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614" exitCode=2
Jul 01 17:22:39.414747 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:39.414724    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerDied","Data":"06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614"}
Jul 01 17:22:42.130669 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:42.130628    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 01 17:22:42.135116 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:42.135086    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 01 17:22:43.849366 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.849341    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:22:43.915785 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.915698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkgsp\" (UniqueName: \"kubernetes.io/projected/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-api-access-zkgsp\") pod \"d5450776-16b9-4d97-8dfc-c77ce263c199\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") "
Jul 01 17:22:43.915785 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.915746    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-rbac-proxy-sar-config\") pod \"d5450776-16b9-4d97-8dfc-c77ce263c199\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") "
Jul 01 17:22:43.915785 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.915770    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5450776-16b9-4d97-8dfc-c77ce263c199-kserve-provision-location\") pod \"d5450776-16b9-4d97-8dfc-c77ce263c199\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") "
Jul 01 17:22:43.916032 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.915804    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-cabundle-cert\") pod \"d5450776-16b9-4d97-8dfc-c77ce263c199\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") "
Jul 01 17:22:43.916032 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.915860    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls\") pod \"d5450776-16b9-4d97-8dfc-c77ce263c199\" (UID: \"d5450776-16b9-4d97-8dfc-c77ce263c199\") "
Jul 01 17:22:43.916148 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.916116    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5450776-16b9-4d97-8dfc-c77ce263c199-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5450776-16b9-4d97-8dfc-c77ce263c199" (UID: "d5450776-16b9-4d97-8dfc-c77ce263c199"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:22:43.916218 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.916189    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5450776-16b9-4d97-8dfc-c77ce263c199" (UID: "d5450776-16b9-4d97-8dfc-c77ce263c199"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:22:43.916327 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.916247    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5450776-16b9-4d97-8dfc-c77ce263c199" (UID: "d5450776-16b9-4d97-8dfc-c77ce263c199"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:22:43.917915 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.917885    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5450776-16b9-4d97-8dfc-c77ce263c199" (UID: "d5450776-16b9-4d97-8dfc-c77ce263c199"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:22:43.918012 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:43.917928    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-api-access-zkgsp" (OuterVolumeSpecName: "kube-api-access-zkgsp") pod "d5450776-16b9-4d97-8dfc-c77ce263c199" (UID: "d5450776-16b9-4d97-8dfc-c77ce263c199"). InnerVolumeSpecName "kube-api-access-zkgsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:22:44.016996 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.016959    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:22:44.016996 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.016989    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5450776-16b9-4d97-8dfc-c77ce263c199-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:22:44.016996 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.017003    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkgsp\" (UniqueName: \"kubernetes.io/projected/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-api-access-zkgsp\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:22:44.017246 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.017015    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5450776-16b9-4d97-8dfc-c77ce263c199-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:22:44.017246 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.017024    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5450776-16b9-4d97-8dfc-c77ce263c199-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:22:44.156193 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156160    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"]
Jul 01 17:22:44.156421 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156409    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kube-rbac-proxy"
Jul 01 17:22:44.156465 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156423    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kube-rbac-proxy"
Jul 01 17:22:44.156465 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156431    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="storage-initializer"
Jul 01 17:22:44.156465 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156436    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="storage-initializer"
Jul 01 17:22:44.156465 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156444    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container"
Jul 01 17:22:44.156465 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156449    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container"
Jul 01 17:22:44.156613 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156491    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kserve-container"
Jul 01 17:22:44.156613 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.156499    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerName="kube-rbac-proxy"
Jul 01 17:22:44.159643 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.159625    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.162609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.162585    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-3b92a-predictor-serving-cert\""
Jul 01 17:22:44.162609 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.162597    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-3b92a-kube-rbac-proxy-sar-config\""
Jul 01 17:22:44.180617 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.180556    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"]
Jul 01 17:22:44.218337 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.218300    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73f389a2-24e4-4812-9bc9-0834ab5927b2-proxy-tls\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.218508 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.218351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.218508 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.218409    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73f389a2-24e4-4812-9bc9-0834ab5927b2-kserve-provision-location\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.218508 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.218474    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78c7s\" (UniqueName: \"kubernetes.io/projected/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-api-access-78c7s\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.218796 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.218509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-cabundle-cert\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.319517 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.319471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.319517 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.319519    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73f389a2-24e4-4812-9bc9-0834ab5927b2-kserve-provision-location\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.319787 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.319566    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c7s\" (UniqueName: \"kubernetes.io/projected/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-api-access-78c7s\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.319787 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.319609    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-cabundle-cert\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.319787 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.319636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73f389a2-24e4-4812-9bc9-0834ab5927b2-proxy-tls\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.320087 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.320034    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73f389a2-24e4-4812-9bc9-0834ab5927b2-kserve-provision-location\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.320257 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.320238    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.320311 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.320238    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-cabundle-cert\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.322079 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.322046    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73f389a2-24e4-4812-9bc9-0834ab5927b2-proxy-tls\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.328777 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.328756    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78c7s\" (UniqueName: \"kubernetes.io/projected/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-api-access-78c7s\") pod \"raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.433673 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.433574    2556 generic.go:356] "Generic (PLEG): container finished" podID="d5450776-16b9-4d97-8dfc-c77ce263c199" containerID="4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44" exitCode=0
Jul 01 17:22:44.433673 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.433649    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerDied","Data":"4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44"}
Jul 01 17:22:44.433673 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.433671    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"
Jul 01 17:22:44.433946 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.433750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss" event={"ID":"d5450776-16b9-4d97-8dfc-c77ce263c199","Type":"ContainerDied","Data":"f601ef7d2f9f41e12d780ca1020a991ec1b6c962204a372fa19159c9021ffed6"}
Jul 01 17:22:44.433946 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.433781    2556 scope.go:117] "RemoveContainer" containerID="06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614"
Jul 01 17:22:44.441479 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.441463    2556 scope.go:117] "RemoveContainer" containerID="4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44"
Jul 01 17:22:44.448464 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.448443    2556 scope.go:117] "RemoveContainer" containerID="89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f"
Jul 01 17:22:44.454211 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.454191    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"]
Jul 01 17:22:44.455690 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.455669    2556 scope.go:117] "RemoveContainer" containerID="06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614"
Jul 01 17:22:44.455950 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:22:44.455927    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614\": container with ID starting with 06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614 not found: ID does not exist" containerID="06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614"
Jul 01 17:22:44.456037 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.455961    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614"} err="failed to get container status \"06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614\": rpc error: code = NotFound desc = could not find container \"06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614\": container with ID starting with 06907d83875ac795abc4311afc8f6dd422aad03361cd75ac42ee8e876f623614 not found: ID does not exist"
Jul 01 17:22:44.456037 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.455986    2556 scope.go:117] "RemoveContainer" containerID="4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44"
Jul 01 17:22:44.456259 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:22:44.456242    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44\": container with ID starting with 4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44 not found: ID does not exist" containerID="4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44"
Jul 01 17:22:44.456298 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.456265    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44"} err="failed to get container status \"4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44\": rpc error: code = NotFound desc = could not find container \"4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44\": container with ID starting with 4f362028367c6e900d10c61619420347260468b2155ebdfe81cec57f9193bc44 not found: ID does not exist"
Jul 01 17:22:44.456298 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.456282    2556 scope.go:117] "RemoveContainer" containerID="89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f"
Jul 01 17:22:44.456501 ip-10-0-141-111 kubenswrapper[2556]: E0701 17:22:44.456483    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f\": container with ID starting with 89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f not found: ID does not exist" containerID="89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f"
Jul 01 17:22:44.456555 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.456508    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f"} err="failed to get container status \"89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f\": rpc error: code = NotFound desc = could not find container \"89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f\": container with ID starting with 89aae511deda3b57d49e40e800e0f7aff8501fd9ada8e5e2f64b69c89b01653f not found: ID does not exist"
Jul 01 17:22:44.462421 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.462400    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-08781-predictor-66bbc8c58-tszss"]
Jul 01 17:22:44.468876 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.468853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:44.604383 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:44.604355    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"]
Jul 01 17:22:44.607709 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:22:44.607685    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f389a2_24e4_4812_9bc9_0834ab5927b2.slice/crio-0f61d87b21a3c1d729fd9a68d801a167efca9df614d67e44a3904250eca1a077 WatchSource:0}: Error finding container 0f61d87b21a3c1d729fd9a68d801a167efca9df614d67e44a3904250eca1a077: Status 404 returned error can't find the container with id 0f61d87b21a3c1d729fd9a68d801a167efca9df614d67e44a3904250eca1a077
Jul 01 17:22:45.439176 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:45.439138    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerStarted","Data":"9b241ef830d7317872f4277107ad3c8e276aae2b5754a948ec24f340e3a93433"}
Jul 01 17:22:45.439576 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:45.439193    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerStarted","Data":"0f61d87b21a3c1d729fd9a68d801a167efca9df614d67e44a3904250eca1a077"}
Jul 01 17:22:46.072462 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:46.072428    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5450776-16b9-4d97-8dfc-c77ce263c199" path="/var/lib/kubelet/pods/d5450776-16b9-4d97-8dfc-c77ce263c199/volumes"
Jul 01 17:22:48.450717 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:48.450684    2556 generic.go:356] "Generic (PLEG): container finished" podID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerID="9b241ef830d7317872f4277107ad3c8e276aae2b5754a948ec24f340e3a93433" exitCode=0
Jul 01 17:22:48.451119 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:48.450758    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerDied","Data":"9b241ef830d7317872f4277107ad3c8e276aae2b5754a948ec24f340e3a93433"}
Jul 01 17:22:49.461858 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:49.461822    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerStarted","Data":"dde3a04daba2e388d614eae80f7286fc26af93bf3d544e478141d4779e64feb4"}
Jul 01 17:22:49.462272 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:49.461904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerStarted","Data":"36ae0e7e8733d35bfebb7f8aeabeeabf614f7f689dd686c87cf156fc2b5fb6b0"}
Jul 01 17:22:49.462272 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:49.462186    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:49.462372 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:49.462348    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:49.463634 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:49.463610    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:22:49.484782 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:49.484733    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podStartSLOduration=1.786222159 podStartE2EDuration="5.484717025s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698494866s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:22:49.482952728 +0000 UTC m=+1459.825293390" watchObservedRunningTime="2026-07-01 17:22:49.484717025 +0000 UTC m=+1459.827057684"
Jul 01 17:22:50.465657 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:50.465623    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:22:55.471650 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:55.471623    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:22:55.472259 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:22:55.472229    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:23:05.472455 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:05.472362    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:23:15.472872 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:15.472827    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:23:25.472299 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:25.472255    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:23:30.655851 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:30.655824    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:23:30.659300 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:30.659279    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:23:35.472303 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:35.472261    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:23:45.472938 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:45.472899    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:23:55.472637 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:23:55.472595    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 01 17:24:02.072474 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:02.072449    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:24:04.083904 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:04.083874    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"]
Jul 01 17:24:04.084433 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:04.084108    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container" containerID="cri-o://36ae0e7e8733d35bfebb7f8aeabeeabf614f7f689dd686c87cf156fc2b5fb6b0" gracePeriod=30
Jul 01 17:24:04.084433 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:04.084410    2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kube-rbac-proxy" containerID="cri-o://dde3a04daba2e388d614eae80f7286fc26af93bf3d544e478141d4779e64feb4" gracePeriod=30
Jul 01 17:24:04.722453 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:04.722417    2556 generic.go:356] "Generic (PLEG): container finished" podID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerID="dde3a04daba2e388d614eae80f7286fc26af93bf3d544e478141d4779e64feb4" exitCode=2
Jul 01 17:24:04.722628 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:04.722486    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerDied","Data":"dde3a04daba2e388d614eae80f7286fc26af93bf3d544e478141d4779e64feb4"}
Jul 01 17:24:05.466224 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:05.466170    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 01 17:24:08.739714 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.739684    2556 generic.go:356] "Generic (PLEG): container finished" podID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerID="36ae0e7e8733d35bfebb7f8aeabeeabf614f7f689dd686c87cf156fc2b5fb6b0" exitCode=0
Jul 01 17:24:08.740084 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.739745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerDied","Data":"36ae0e7e8733d35bfebb7f8aeabeeabf614f7f689dd686c87cf156fc2b5fb6b0"}
Jul 01 17:24:08.837079 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.837034    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:24:08.934769 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.934678    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73f389a2-24e4-4812-9bc9-0834ab5927b2-kserve-provision-location\") pod \"73f389a2-24e4-4812-9bc9-0834ab5927b2\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") "
Jul 01 17:24:08.934769 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.934725    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73f389a2-24e4-4812-9bc9-0834ab5927b2-proxy-tls\") pod \"73f389a2-24e4-4812-9bc9-0834ab5927b2\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") "
Jul 01 17:24:08.934769 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.934757    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78c7s\" (UniqueName: \"kubernetes.io/projected/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-api-access-78c7s\") pod \"73f389a2-24e4-4812-9bc9-0834ab5927b2\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") "
Jul 01 17:24:08.935085 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.934784    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-cabundle-cert\") pod \"73f389a2-24e4-4812-9bc9-0834ab5927b2\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") "
Jul 01 17:24:08.935085 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.934900    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-rbac-proxy-sar-config\") pod \"73f389a2-24e4-4812-9bc9-0834ab5927b2\" (UID: \"73f389a2-24e4-4812-9bc9-0834ab5927b2\") "
Jul 01 17:24:08.935196 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.935093    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f389a2-24e4-4812-9bc9-0834ab5927b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73f389a2-24e4-4812-9bc9-0834ab5927b2" (UID: "73f389a2-24e4-4812-9bc9-0834ab5927b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:24:08.935264 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.935212    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "73f389a2-24e4-4812-9bc9-0834ab5927b2" (UID: "73f389a2-24e4-4812-9bc9-0834ab5927b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:24:08.935307 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.935287    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "73f389a2-24e4-4812-9bc9-0834ab5927b2" (UID: "73f389a2-24e4-4812-9bc9-0834ab5927b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:24:08.936749 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.936725    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f389a2-24e4-4812-9bc9-0834ab5927b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73f389a2-24e4-4812-9bc9-0834ab5927b2" (UID: "73f389a2-24e4-4812-9bc9-0834ab5927b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:24:08.936823 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:08.936773    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-api-access-78c7s" (OuterVolumeSpecName: "kube-api-access-78c7s") pod "73f389a2-24e4-4812-9bc9-0834ab5927b2" (UID: "73f389a2-24e4-4812-9bc9-0834ab5927b2"). InnerVolumeSpecName "kube-api-access-78c7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:24:09.035806 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.035770    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73f389a2-24e4-4812-9bc9-0834ab5927b2-kserve-provision-location\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:24:09.035806 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.035796    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73f389a2-24e4-4812-9bc9-0834ab5927b2-proxy-tls\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:24:09.035806 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.035807    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78c7s\" (UniqueName: \"kubernetes.io/projected/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-api-access-78c7s\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:24:09.035806 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.035815    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-cabundle-cert\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:24:09.036052 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.035823    2556 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73f389a2-24e4-4812-9bc9-0834ab5927b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-111.ec2.internal\" DevicePath \"\""
Jul 01 17:24:09.745205 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.745161    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl" event={"ID":"73f389a2-24e4-4812-9bc9-0834ab5927b2","Type":"ContainerDied","Data":"0f61d87b21a3c1d729fd9a68d801a167efca9df614d67e44a3904250eca1a077"}
Jul 01 17:24:09.745662 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.745253    2556 scope.go:117] "RemoveContainer" containerID="dde3a04daba2e388d614eae80f7286fc26af93bf3d544e478141d4779e64feb4"
Jul 01 17:24:09.745662 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.745255    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"
Jul 01 17:24:09.754189 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.754168    2556 scope.go:117] "RemoveContainer" containerID="36ae0e7e8733d35bfebb7f8aeabeeabf614f7f689dd686c87cf156fc2b5fb6b0"
Jul 01 17:24:09.761601 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.761583    2556 scope.go:117] "RemoveContainer" containerID="9b241ef830d7317872f4277107ad3c8e276aae2b5754a948ec24f340e3a93433"
Jul 01 17:24:09.777136 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.777112    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"]
Jul 01 17:24:09.787328 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:09.787301    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-3b92a-predictor-74df696594-d49cl"]
Jul 01 17:24:10.073980 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:10.073901    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" path="/var/lib/kubelet/pods/73f389a2-24e4-4812-9bc9-0834ab5927b2/volumes"
Jul 01 17:24:40.372005 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.371973    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crfk4/must-gather-jgx9m"]
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372237    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kube-rbac-proxy"
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372249    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kube-rbac-proxy"
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372260    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="storage-initializer"
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372266    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="storage-initializer"
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372274    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container"
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372279    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container"
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372319    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kserve-container"
Jul 01 17:24:40.372429 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.372327    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f389a2-24e4-4812-9bc9-0834ab5927b2" containerName="kube-rbac-proxy"
Jul 01 17:24:40.376170 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.376153    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.378968 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.378949    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crfk4\"/\"kube-root-ca.crt\""
Jul 01 17:24:40.379124 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.378950    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-crfk4\"/\"openshift-service-ca.crt\""
Jul 01 17:24:40.379973 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.379956    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-crfk4\"/\"default-dockercfg-s5btw\""
Jul 01 17:24:40.384722 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.384692    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crfk4/must-gather-jgx9m"]
Jul 01 17:24:40.462756 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.462723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lft8n\" (UniqueName: \"kubernetes.io/projected/6257fa8d-7111-41c4-b102-747554f06cad-kube-api-access-lft8n\") pod \"must-gather-jgx9m\" (UID: \"6257fa8d-7111-41c4-b102-747554f06cad\") " pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.462942 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.462766    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6257fa8d-7111-41c4-b102-747554f06cad-must-gather-output\") pod \"must-gather-jgx9m\" (UID: \"6257fa8d-7111-41c4-b102-747554f06cad\") " pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.563254 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.563207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6257fa8d-7111-41c4-b102-747554f06cad-must-gather-output\") pod \"must-gather-jgx9m\" (UID: \"6257fa8d-7111-41c4-b102-747554f06cad\") " pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.563435 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.563273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lft8n\" (UniqueName: \"kubernetes.io/projected/6257fa8d-7111-41c4-b102-747554f06cad-kube-api-access-lft8n\") pod \"must-gather-jgx9m\" (UID: \"6257fa8d-7111-41c4-b102-747554f06cad\") " pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.563565 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.563546    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6257fa8d-7111-41c4-b102-747554f06cad-must-gather-output\") pod \"must-gather-jgx9m\" (UID: \"6257fa8d-7111-41c4-b102-747554f06cad\") " pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.571447 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.571420    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lft8n\" (UniqueName: \"kubernetes.io/projected/6257fa8d-7111-41c4-b102-747554f06cad-kube-api-access-lft8n\") pod \"must-gather-jgx9m\" (UID: \"6257fa8d-7111-41c4-b102-747554f06cad\") " pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.685410 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.685327    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crfk4/must-gather-jgx9m"
Jul 01 17:24:40.811809 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.811782    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crfk4/must-gather-jgx9m"]
Jul 01 17:24:40.814445 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:24:40.814418    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6257fa8d_7111_41c4_b102_747554f06cad.slice/crio-c3a57cb69c30c4bcf4bc7a21ec34902799d15cf9cce5ac582346405fbe786d9a WatchSource:0}: Error finding container c3a57cb69c30c4bcf4bc7a21ec34902799d15cf9cce5ac582346405fbe786d9a: Status 404 returned error can't find the container with id c3a57cb69c30c4bcf4bc7a21ec34902799d15cf9cce5ac582346405fbe786d9a
Jul 01 17:24:40.816672 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.816655    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:24:40.853069 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:40.853027    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crfk4/must-gather-jgx9m" event={"ID":"6257fa8d-7111-41c4-b102-747554f06cad","Type":"ContainerStarted","Data":"c3a57cb69c30c4bcf4bc7a21ec34902799d15cf9cce5ac582346405fbe786d9a"}
Jul 01 17:24:42.863794 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:42.863750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crfk4/must-gather-jgx9m" event={"ID":"6257fa8d-7111-41c4-b102-747554f06cad","Type":"ContainerStarted","Data":"8f2bb61e3590292e31f026f4a1fb55c210d90af8e9eb99d11df4d55b307e74a2"}
Jul 01 17:24:42.863794 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:42.863798    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crfk4/must-gather-jgx9m" event={"ID":"6257fa8d-7111-41c4-b102-747554f06cad","Type":"ContainerStarted","Data":"cb7bb4b208f21d69c0b8449362a0b4412d409f069610b38e62c87aadc47a007f"}
Jul 01 17:24:42.890321 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:42.890260    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crfk4/must-gather-jgx9m" podStartSLOduration=1.421918156 podStartE2EDuration="2.890241132s" totalImagesPullingTime="1.468322976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:24:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:24:42.889126198 +0000 UTC m=+1573.231466858" watchObservedRunningTime="2026-07-01 17:24:42.890241132 +0000 UTC m=+1573.232581793"
Jul 01 17:24:43.831198 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:43.831168    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-npk7n_40127afb-5366-4e79-9f41-64186c92e106/global-pull-secret-syncer/0.log"
Jul 01 17:24:43.926532 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:43.926499    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ddgms_b8e605ac-da6b-40ed-8fa6-079d387ea570/konnectivity-agent/0.log"
Jul 01 17:24:44.051219 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:44.051185    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-111.ec2.internal_80ecdb78f14d99528a812fe67731e2c2/haproxy/0.log"
Jul 01 17:24:48.042086 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:48.042031    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-54cb84584-44p68_fb1e46ce-6d0c-4bcd-b895-01f74c6a829f/cluster-monitoring-operator/0.log"
Jul 01 17:24:48.225659 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:48.225547    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qkpl_dd6f2e1e-9296-4ed9-9250-888376d6f734/node-exporter/0.log"
Jul 01 17:24:48.255374 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:48.255348    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qkpl_dd6f2e1e-9296-4ed9-9250-888376d6f734/kube-rbac-proxy/0.log"
Jul 01 17:24:48.284656 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:48.284618    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qkpl_dd6f2e1e-9296-4ed9-9250-888376d6f734/init-textfile/0.log"
Jul 01 17:24:51.981933 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:51.981904    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-69d9f679db-hddrd_b4c671d8-43bd-4a42-a885-81d70371a7d3/volume-data-source-validator/0.log"
Jul 01 17:24:52.222188 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.222153    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"]
Jul 01 17:24:52.227415 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.227386    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.238938 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.238908    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"]
Jul 01 17:24:52.361374 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.361331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-podres\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.361558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.361404    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-proc\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.361558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.361432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6r7\" (UniqueName: \"kubernetes.io/projected/b312bf58-81c1-43b9-965d-5c8c10004cd6-kube-api-access-6k6r7\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.361558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.361472    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-sys\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.361558 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.361514    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-lib-modules\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462126 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-podres\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462310 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-proc\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462310 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6r7\" (UniqueName: \"kubernetes.io/projected/b312bf58-81c1-43b9-965d-5c8c10004cd6-kube-api-access-6k6r7\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462310 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-sys\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462310 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-lib-modules\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462527 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462465    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-lib-modules\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462581 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462545    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-podres\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462635 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462585    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-proc\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.462954 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.462931    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b312bf58-81c1-43b9-965d-5c8c10004cd6-sys\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.475422 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.475364    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6r7\" (UniqueName: \"kubernetes.io/projected/b312bf58-81c1-43b9-965d-5c8c10004cd6-kube-api-access-6k6r7\") pod \"perf-node-gather-daemonset-qlz6n\" (UID: \"b312bf58-81c1-43b9-965d-5c8c10004cd6\") " pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.541607 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.541464    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.713450 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.713312    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"]
Jul 01 17:24:52.716469 ip-10-0-141-111 kubenswrapper[2556]: W0701 17:24:52.716428    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb312bf58_81c1_43b9_965d_5c8c10004cd6.slice/crio-89bc5810d760434dea14cbbd46f09cdc350bbd2d3e68b800d153f516f735ab69 WatchSource:0}: Error finding container 89bc5810d760434dea14cbbd46f09cdc350bbd2d3e68b800d153f516f735ab69: Status 404 returned error can't find the container with id 89bc5810d760434dea14cbbd46f09cdc350bbd2d3e68b800d153f516f735ab69
Jul 01 17:24:52.920172 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.920128    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n" event={"ID":"b312bf58-81c1-43b9-965d-5c8c10004cd6","Type":"ContainerStarted","Data":"a8f295a70563e7647d7d84183c7c6df73e6855fcae2e148200c50e7a8162afdb"}
Jul 01 17:24:52.920364 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.920195    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n" event={"ID":"b312bf58-81c1-43b9-965d-5c8c10004cd6","Type":"ContainerStarted","Data":"89bc5810d760434dea14cbbd46f09cdc350bbd2d3e68b800d153f516f735ab69"}
Jul 01 17:24:52.920432 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.920372    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:24:52.939338 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:52.939275    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n" podStartSLOduration=0.939255735 podStartE2EDuration="939.255735ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:24:52.937781741 +0000 UTC m=+1583.280122402" watchObservedRunningTime="2026-07-01 17:24:52.939255735 +0000 UTC m=+1583.281596395"
Jul 01 17:24:53.070510 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:53.070479    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vx286_af5c8513-b418-4256-8a57-6cf1381096b9/dns/0.log"
Jul 01 17:24:53.093312 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:53.093276    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vx286_af5c8513-b418-4256-8a57-6cf1381096b9/kube-rbac-proxy/0.log"
Jul 01 17:24:53.143713 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:53.143647    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hq7vh_b277c8b7-1860-4ca6-97db-9e64e6f2ec9d/dns-node-resolver/0.log"
Jul 01 17:24:53.823187 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:53.823160    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f6pw8_1f0b617c-0bf7-4462-a62b-2aee12c53d78/node-ca/0.log"
Jul 01 17:24:54.722542 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:54.722513    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79bb6d4bff-plxzt_6dd03f61-baea-420b-bf09-060e6338f8f4/router/0.log"
Jul 01 17:24:55.143292 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:55.143261    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6jjk_b8e0b3b7-2505-4bdb-bf4f-285c943fece3/serve-healthcheck-canary/0.log"
Jul 01 17:24:55.565801 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:55.565777    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bl5g_c9712d61-9858-4ec0-ae38-e3e16e9fa2d9/kube-rbac-proxy/0.log"
Jul 01 17:24:55.594672 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:55.594649    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bl5g_c9712d61-9858-4ec0-ae38-e3e16e9fa2d9/exporter/0.log"
Jul 01 17:24:55.620202 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:55.620170    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bl5g_c9712d61-9858-4ec0-ae38-e3e16e9fa2d9/extractor/0.log"
Jul 01 17:24:57.908431 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:57.908375    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-67d95fdc48-96gp7_a3a925f2-ba56-47ba-9801-b0aeab3191c8/manager/0.log"
Jul 01 17:24:58.028029 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:58.027996    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-xrpwl_8ff7439b-31b9-452d-9f44-1e0a3bea8ca5/manager/0.log"
Jul 01 17:24:58.117526 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:58.117481    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-86fpm_750af89d-bda4-4168-b871-194dc163c6d3/seaweedfs/0.log"
Jul 01 17:24:58.935589 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:24:58.935563    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-crfk4/perf-node-gather-daemonset-qlz6n"
Jul 01 17:25:03.750732 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:03.750703    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwnqp_c53c6577-4cf4-45a2-802a-690ea2560a57/kube-multus-additional-cni-plugins/0.log"
Jul 01 17:25:03.777803 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:03.777778    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwnqp_c53c6577-4cf4-45a2-802a-690ea2560a57/egress-router-binary-copy/0.log"
Jul 01 17:25:03.802041 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:03.802020    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwnqp_c53c6577-4cf4-45a2-802a-690ea2560a57/cni-plugins/0.log"
Jul 01 17:25:03.826862 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:03.826833    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwnqp_c53c6577-4cf4-45a2-802a-690ea2560a57/bond-cni-plugin/0.log"
Jul 01 17:25:03.850215 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:03.850190    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwnqp_c53c6577-4cf4-45a2-802a-690ea2560a57/routeoverride-cni/0.log"
Jul 01 17:25:03.875450 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:03.875421    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwnqp_c53c6577-4cf4-45a2-802a-690ea2560a57/whereabouts-cni-bincopy/0.log"
Jul 01 17:25:03.898523 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:03.898502    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwnqp_c53c6577-4cf4-45a2-802a-690ea2560a57/whereabouts-cni/0.log"
Jul 01 17:25:04.300105 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:04.300073    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c8c6g_d271da50-b129-4761-9ece-af5b472a97e8/kube-multus/0.log"
Jul 01 17:25:04.361094 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:04.361055    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-chkmv_406c4f26-829c-4bbc-b670-2adde865c36f/network-metrics-daemon/0.log"
Jul 01 17:25:04.391614 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:04.391578    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-chkmv_406c4f26-829c-4bbc-b670-2adde865c36f/kube-rbac-proxy/0.log"
Jul 01 17:25:05.251236 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.251206    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-controller/0.log"
Jul 01 17:25:05.275189 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.275160    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/1.log"
Jul 01 17:25:05.275189 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.275160    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovn-acl-logging/0.log"
Jul 01 17:25:05.293937 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.293897    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/kube-rbac-proxy-node/0.log"
Jul 01 17:25:05.317748 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.317707    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 17:25:05.341695 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.341664    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/northd/0.log"
Jul 01 17:25:05.369904 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.369880    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/nbdb/0.log"
Jul 01 17:25:05.392320 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.392276    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/sbdb/0.log"
Jul 01 17:25:05.435685 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:05.435650    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cht4l_067f479a-c3e1-4a2e-badc-f863f901a14b/ovnkube-controller/0.log"
Jul 01 17:25:06.914732 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:06.914706    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rkmmg_0f425d31-4437-4e8d-bd20-3c3e40bdb24b/network-check-target-container/0.log"
Jul 01 17:25:07.991352 ip-10-0-141-111 kubenswrapper[2556]: I0701 17:25:07.991326    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6phpp_f18c7204-9546-4e24-ae93-4edb51af02dd/iptables-alerter/0.log"