Jul 07 11:38:03.229612 ip-10-0-143-227 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 11:38:03.229624 ip-10-0-143-227 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 11:38:03.229631 ip-10-0-143-227 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 11:38:03.230033 ip-10-0-143-227 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 11:38:13.320904 ip-10-0-143-227 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 11:38:13.320920 ip-10-0-143-227 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 65297b12a560451fb3af3a6ac7b9c72e --
Jul 07 11:40:13.994523 ip-10-0-143-227 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 11:40:14.407869 ip-10-0-143-227 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:14.407869 ip-10-0-143-227 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 11:40:14.407869 ip-10-0-143-227 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:14.407869 ip-10-0-143-227 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 11:40:14.407869 ip-10-0-143-227 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 11:40:14.409676 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.409511    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 11:40:14.417799 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417775    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.417799 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417794    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.417799 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417798    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.417799 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417802    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.417799 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417805    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.417799 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417808    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417811    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417814    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417817    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417819    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417822    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417825    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417827    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417830    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417832    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417835    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417837    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417840    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417843    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417845    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417848    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417853    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417857    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417859    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.418023 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417863    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417867    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417870    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417872    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417875    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417878    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417880    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417883    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417893    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417895    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417898    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417900    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417903    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417906    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417908    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417911    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417914    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417917    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417919    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417922    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.418523 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417924    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417927    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417929    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417932    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417935    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417937    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417940    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417943    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417945    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417948    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417951    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417953    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417956    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417958    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417960    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417963    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417965    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417968    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417970    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417973    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.419002 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417975    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417977    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417980    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417983    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417986    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417988    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.417992    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418000    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418003    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418005    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418008    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418010    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418013    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418015    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418018    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418020    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418023    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418025    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418028    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.419503 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418031    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418034    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418036    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418039    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418042    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418044    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418046    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418049    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418052    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418054    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418057    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418060    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418062    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418064    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418067    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.418069    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419901    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419910    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419913    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.420003 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419916    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419920    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419924    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419927    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419930    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419932    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419935    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419938    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419940    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419943    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419945    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419948    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419950    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419953    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419956    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419959    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419962    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419965    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419967    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419970    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.420481 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419972    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419975    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419977    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419980    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419983    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419985    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419988    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419991    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419995    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.419998    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420000    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420003    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420006    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420008    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420011    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420014    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420017    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420019    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420022    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.421079 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420024    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420027    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420029    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420032    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420035    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420037    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420039    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420042    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420047    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420050    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420052    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420055    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420057    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420059    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420062    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420064    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420067    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420070    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420073    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420076    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.421554 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420078    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420081    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420083    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420085    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420088    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420091    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420093    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420096    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420098    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420100    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420103    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420105    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420108    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420110    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420113    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420115    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420118    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420120    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420123    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.422044 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420127    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420132    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420135    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420138    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420141    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420144    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420146    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420149    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420151    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420154    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420156    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420159    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420161    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420164    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420166    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420169    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420172    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420174    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420584    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.422539 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420590    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420594    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420597    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420600    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420603    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420606    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420609    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420611    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420614    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420616    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420619    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420623    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420627    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420629    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420634    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420638    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420641    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420644    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.423007 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420647    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420650    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420653    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420655    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420658    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420660    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420663    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420666    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420668    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420670    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420673    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420675    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420678    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420681    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420683    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420686    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420688    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420691    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420693    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420696    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.423533 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420698    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420700    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420703    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420705    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420708    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420710    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420712    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420715    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420718    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420721    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420724    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420726    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420729    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420732    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420736    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420738    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420741    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420743    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420746    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420748    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.424016 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420751    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420753    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420756    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420758    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420761    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420763    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420765    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420768    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420770    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420773    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420775    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420778    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420780    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420783    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420785    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420787    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420790    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420794    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420797    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420799    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.424525 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420802    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420805    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420807    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420811    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420813    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420816    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420818    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420821    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420825    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420828    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420831    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420834    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420836    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420839    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420841    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420844    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420847    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420849    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420852    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.420854    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.425039 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420922    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420929    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420943    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420952    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420956    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420959    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420963    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420967    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420970    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420973    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420976    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420979    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420987    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420992    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.420996    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421000    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421005    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421008    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421012    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421015    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421018    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421021    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421025    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421028    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 11:40:14.425538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421031    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421034    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421037    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421040    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421043    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421046    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421049    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421052    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421056    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421060    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421062    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421066    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421069    2565 flags.go:64] FLAG: --enable-server="true"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421072    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421076    2565 flags.go:64] FLAG: --event-burst="100"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421080    2565 flags.go:64] FLAG: --event-qps="50"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421083    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421086    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421089    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421092    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421097    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421100    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421103    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421106    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421109    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 11:40:14.426120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421112    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421115    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421118    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421120    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421123    2565 flags.go:64] FLAG: --feature-gates=""
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421127    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421130    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421133    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421136    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421139    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421141    2565 flags.go:64] FLAG: --help="false"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421144    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421147    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421150    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421153    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421156    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421159    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421162    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421165    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421168    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421171    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421173    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421176    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421179    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 07 11:40:14.426765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421182    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421185    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421188    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421193    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421196    2565 flags.go:64] FLAG: --lock-file=""
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421198    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421201    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421204    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421209    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421213    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421234    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421238    2565 flags.go:64] FLAG: --logging-format="text"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421241    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421245    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421248    2565 flags.go:64] FLAG: --manifest-url=""
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421250    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421255    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421258    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421262    2565 flags.go:64] FLAG: --max-pods="110"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421265    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421268    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421271    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421274    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421276    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421279    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 11:40:14.427750 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421282    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421289    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421292    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421295    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421298    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421301    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421307    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421309    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421313    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421315    2565 flags.go:64] FLAG: --port="10250"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421320    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421324    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e967341b4aa0b96a"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421327    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421330    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421333    2565 flags.go:64] FLAG: --register-node="true"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421336    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421340    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421343    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421346    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421349    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421352    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421355    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421358    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421361    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421364    2565 flags.go:64] FLAG: --runonce="false"
Jul 07 11:40:14.428822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421366    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421369    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421372    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421375    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421378    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421380    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421383    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421386    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421389    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421392    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421395    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421397    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421400    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421403    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421406    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421412    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421415    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421418    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421424    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421427    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421430    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421433    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421435    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421438    2565 flags.go:64] FLAG: --v="2"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421442    2565 flags.go:64] FLAG: --version="false"
Jul 07 11:40:14.429959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421446    2565 flags.go:64] FLAG: --vmodule=""
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421450    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421453    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421564    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421567    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421570    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421572    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421575    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421578    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421581    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421584    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421586    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421589    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421591    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421594    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421596    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421599    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421601    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421604    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421606    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421609    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.431060 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421611    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421614    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421617    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421620    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421625    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421627    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421630    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421632    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421635    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421638    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421641    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421643    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421647    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421649    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421652    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421654    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421657    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421659    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421662    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421664    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.431664 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421666    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421669    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421671    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421674    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421676    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421678    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421681    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421683    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421686    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421688    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421691    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421693    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421696    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421698    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421701    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421703    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421707    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421709    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421713    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.432499 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421717    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421720    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421723    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421726    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421729    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421732    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421737    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421740    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421743    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421746    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421749    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421751    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421753    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421756    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421758    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421761    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421763    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421766    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421768    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421770    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.433269 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421773    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421775    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421778    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421780    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421783    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421785    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421788    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421790    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421793    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421797    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421800    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421802    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421805    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421807    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421810    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421813    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421815    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421819    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421822    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.433846 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421825    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421828    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.421831    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.421836    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.429519    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.429541    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.429567    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.429576    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429668    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429676    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429682    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429686    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429691    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429696    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429700    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.434379 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429705    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429710    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429715    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429719    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429723    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429728    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429732    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429737    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429741    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429745    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429749    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429753    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429757    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429764    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429770    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429775    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429780    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429785    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429790    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.434771 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429794    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429798    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429803    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429809    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429814    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429818    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429822    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429827    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429831    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429835    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429839    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429844    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429848    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429852    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429856    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429860    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429864    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429869    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429873    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429877    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.435339 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429881    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429887    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429891    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429895    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429899    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429903    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429908    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429912    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429916    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429920    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429924    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429929    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429933    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429937    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429941    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429947    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429952    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429956    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429960    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.435850 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429966    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429971    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429975    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429980    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429984    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429989    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429993    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.429997    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430002    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430006    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430010    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430014    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430018    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430022    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430026    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430030    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430035    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430039    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430043    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430047    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.436341 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430052    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430056    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430060    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430064    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430068    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430072    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430077    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430080    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430085    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430092    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430098    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430103    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430107    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430111    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.430119    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:14.436871 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430345    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430354    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430358    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430363    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430367    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430372    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430376    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430380    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430385    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430388    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430392    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430396    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430400    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430404    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430409    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430413    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430417    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430421    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430425    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430430    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 11:40:14.437265 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430434    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430438    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430442    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430446    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430451    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430456    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430460    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430465    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430470    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430474    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430479    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430483    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430487    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430491    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430495    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430499    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430503    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430507    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430512    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430516    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 11:40:14.437758 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430520    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430524    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430528    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430532    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430537    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430541    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430547    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430553    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430559    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430564    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430591    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430597    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430603    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430607    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430612    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430618    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430622    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430628    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430632    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 11:40:14.438278 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430637    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430641    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430647    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430652    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430656    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430661    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430665    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430669    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430673    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430677    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430681    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430686    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430703    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430710    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430715    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430722    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430728    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430733    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430738    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 11:40:14.438737 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430742    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430746    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430751    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430755    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430759    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430763    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430767    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430771    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430775    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430779    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430785    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430791    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430796    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430801    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430806    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430810    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430815    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430819    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430823    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 11:40:14.439189 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430827    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:14.430831    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.430839    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.430985    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.433725    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.435250    2565 server.go:1019] "Starting client certificate rotation"
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.435352    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.435380    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 11:40:14.439896 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.438410    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 11:40:14.444450 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.444421    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 11:40:14.457628 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.457610    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 07 11:40:14.459731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.459716    2565 log.go:25] "Validated CRI v1 image API"
Jul 07 11:40:14.460551 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.460529    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 11:40:14.462587 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.462570    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 11:40:14.463632 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.463611    2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 885eb7a4-c89a-4a21-9633-e5f1ed8b879a:/dev/nvme0n1p4 a25e39b0-9c61-4492-8118-c9e2dfb72ba4:/dev/nvme0n1p3]
Jul 07 11:40:14.463685 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.463632    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 11:40:14.468557 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.468441    2565 manager.go:216] Machine: {Timestamp:2026-07-07 11:40:14.467250799 +0000 UTC m=+0.357444839 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100265 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27cf81ce23be5db5e5dc093e45f0dc SystemUUID:ec27cf81-ce23-be5d-b5e5-dc093e45f0dc BootID:65297b12-a560-451f-b3af-3a6ac7b9c72e Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:aa:17:6e:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:aa:17:6e:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:12:83:5c:aa:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 11:40:14.468557 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.468551    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 11:40:14.468671 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.468633    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 11:40:14.470852 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.470829    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 11:40:14.470984 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.470854    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 11:40:14.471582 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.471572    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 11:40:14.471613 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.471585    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 11:40:14.471613 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.471592    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 11:40:14.472262 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.472252    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 11:40:14.473565 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.473556    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 11:40:14.476629 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.476618    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 11:40:14.476743 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.476734    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 11:40:14.476773 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.476763    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 11:40:14.476773 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.476772    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 11:40:14.476829 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.476785    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 11:40:14.476829 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.476795    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 07 11:40:14.476829 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.476804    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 11:40:14.477176 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.477162    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 11:40:14.477252 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.477179    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 11:40:14.481118 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.481101    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 11:40:14.482960 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.482945    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 11:40:14.483033 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.482975    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 11:40:14.484236 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484211    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 11:40:14.484279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484242    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 11:40:14.484279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484248    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 11:40:14.484279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484253    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 11:40:14.484279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484262    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 11:40:14.484279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484268    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 11:40:14.484279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484276    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 11:40:14.484279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484281    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 11:40:14.484462 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484288    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 11:40:14.484462 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484294    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 11:40:14.484462 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484304    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 11:40:14.484462 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484310    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 11:40:14.484971 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484961    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 11:40:14.484971 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.484970    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 11:40:14.489945 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.489922    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 11:40:14.490013 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.489944    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 11:40:14.490013 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.489942    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 11:40:14.490330 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.490318    2565 server.go:1268] "Started kubelet"
Jul 07 11:40:14.490403 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.490374    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 11:40:14.490474 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.490397    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 11:40:14.490521 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.490496    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 11:40:14.491185 ip-10-0-143-227 systemd[1]: Started Kubernetes Kubelet.
Jul 07 11:40:14.492370 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.492357    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 11:40:14.494725 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.494706    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 11:40:14.496819 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.495743    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-227.ec2.internal.18bfff0e5573cafc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-227.ec2.internal,UID:ip-10-0-143-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-227.ec2.internal,},FirstTimestamp:2026-07-07 11:40:14.490282748 +0000 UTC m=+0.380476787,LastTimestamp:2026-07-07 11:40:14.490282748 +0000 UTC m=+0.380476787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-227.ec2.internal,}"
Jul 07 11:40:14.498108 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.498081    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 11:40:14.498939 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.498925    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pvq6"
Jul 07 11:40:14.501948 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.501929    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 11:40:14.501948 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.501938    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 11:40:14.502084 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.502026    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:14.502631 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.502613    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 11:40:14.502719 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.502634    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 11:40:14.502719 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.502614    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 11:40:14.502719 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.502718    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 11:40:14.502845 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.502727    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 11:40:14.502845 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.502799    2565 factory.go:55] Registering systemd factory
Jul 07 11:40:14.502845 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.502815    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 07 11:40:14.503070 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.503058    2565 factory.go:153] Registering CRI-O factory
Jul 07 11:40:14.503126 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.503073    2565 factory.go:223] Registration of the crio container factory successfully
Jul 07 11:40:14.503126 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.503122    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 11:40:14.503232 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.503146    2565 factory.go:103] Registering Raw factory
Jul 07 11:40:14.503232 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.503160    2565 manager.go:1196] Started watching for new ooms in manager
Jul 07 11:40:14.503754 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.503624    2565 manager.go:349] Starting recovery of all containers
Jul 07 11:40:14.504827 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.504804    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 11:40:14.504908 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.504875    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 11:40:14.507643 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.507617    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pvq6"
Jul 07 11:40:14.513853 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.513839    2565 manager.go:354] Recovery completed
Jul 07 11:40:14.518777 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.518753    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.520503 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.520486    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.520576 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.520515    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.520576 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.520526    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.521083 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.521066    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 11:40:14.521152 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.521084    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 11:40:14.521152 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.521102    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 11:40:14.521235 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.521099    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.521887 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.521871    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.521964 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.521900    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.521964 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.521915    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.522299 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.522284    2565 policy_none.go:49] "None policy: Start"
Jul 07 11:40:14.522367 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.522304    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 11:40:14.522367 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.522321    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 11:40:14.523432 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.523421    2565 policy_none.go:47] "Start"
Jul 07 11:40:14.561923 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.561733    2565 manager.go:337] "Starting Device Plugin manager"
Jul 07 11:40:14.562042 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.561950    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 11:40:14.562042 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.561966    2565 server.go:87] "Starting device plugin registration server"
Jul 07 11:40:14.562283 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.562270    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 11:40:14.562355 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.562286    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 11:40:14.562407 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.562385    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 11:40:14.562488 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.562470    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 11:40:14.562488 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.562485    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 11:40:14.563009 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.562990    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 11:40:14.563073 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.563033    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:14.662762 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.662699    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.663847 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.663627    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.663847 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.663661    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.663847 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.663671    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.663847 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.663700    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.673298 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.673279    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.673341 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.673303    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-227.ec2.internal\": node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:14.688675 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.688647    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 11:40:14.689714 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.689693    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:14.689849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.689835    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 11:40:14.689931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.689879    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 11:40:14.689931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.689906    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 11:40:14.690030 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.689944    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 11:40:14.691612 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.691596    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:14.790586 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.790546    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"]
Jul 07 11:40:14.790586 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.790566    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:14.790830 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.790637    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.791721 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.791701    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.791822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.791741    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.791822 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.791757    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.793043 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793031    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.793150 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.793204 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793195    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.793812 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793793    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.793890 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793820    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.793890 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793801    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.793890 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793836    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.793890 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793852    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.793890 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.793865    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.795032 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.795015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.795128 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.795045    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 11:40:14.795659 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.795646    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 11:40:14.795738 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.795675    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 11:40:14.795738 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.795688    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeHasSufficientPID"
Jul 07 11:40:14.803957 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.803940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b32639e909a677f36a8e629fed59bab2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"b32639e909a677f36a8e629fed59bab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.804037 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.803969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b32639e909a677f36a8e629fed59bab2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"b32639e909a677f36a8e629fed59bab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.804037 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.803991    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e90e1b7edb9e4ce00addd9710b3030b5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-227.ec2.internal\" (UID: \"e90e1b7edb9e4ce00addd9710b3030b5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.817885 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.817865    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-227.ec2.internal\" not found" node="ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.822244 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.822208    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-227.ec2.internal\" not found" node="ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.890838 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.890816    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:14.904380 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.904358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b32639e909a677f36a8e629fed59bab2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"b32639e909a677f36a8e629fed59bab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.904448 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.904384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b32639e909a677f36a8e629fed59bab2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"b32639e909a677f36a8e629fed59bab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.904448 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.904411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e90e1b7edb9e4ce00addd9710b3030b5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-227.ec2.internal\" (UID: \"e90e1b7edb9e4ce00addd9710b3030b5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.904522 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.904451    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b32639e909a677f36a8e629fed59bab2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"b32639e909a677f36a8e629fed59bab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.904522 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.904457    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b32639e909a677f36a8e629fed59bab2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal\" (UID: \"b32639e909a677f36a8e629fed59bab2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.904522 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:14.904502    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e90e1b7edb9e4ce00addd9710b3030b5-config\") pod \"kube-apiserver-proxy-ip-10-0-143-227.ec2.internal\" (UID: \"e90e1b7edb9e4ce00addd9710b3030b5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:14.991183 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:14.991117    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:15.091699 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.091673    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:15.121131 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.121096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:15.126214 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.126194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:15.192077 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.192048    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:15.292709 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.292637    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:15.393157 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.393121    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-227.ec2.internal\" not found"
Jul 07 11:40:15.434850 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.434824    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 11:40:15.435278 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.434971    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 11:40:15.451051 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.451028    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:15.477573 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.477556    2565 apiserver.go:52] "Watching apiserver"
Jul 07 11:40:15.477991 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.477975    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:15.487276 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.487259    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 11:40:15.487573 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.487555    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-697hj","openshift-network-diagnostics/network-check-target-82drw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd","openshift-dns/node-resolver-vcqtg","openshift-image-registry/node-ca-5h8h7","openshift-multus/multus-additional-cni-plugins-bv5rp","openshift-multus/network-metrics-daemon-lrftd","openshift-network-operator/iptables-alerter-jb4xk","openshift-ovn-kubernetes/ovnkube-node-l5kdk","kube-system/global-pull-secret-syncer-85pvm","kube-system/konnectivity-agent-pd5p5","openshift-cluster-node-tuning-operator/tuned-sbkwj"]
Jul 07 11:40:15.489151 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.489130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.490279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.490259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:15.490356 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.490318    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:15.491556 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.491541    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.492662 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.492646    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.492855 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.492840    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 11:40:15.492904 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.492842    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 11:40:15.493559 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.493544    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 11:40:15.493881 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.493865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.493959 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.493938    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 11:40:15.494940 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.494911    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 11:40:15.495063 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.495043    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 11:40:15.495632 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.495610    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m4m9z\""
Jul 07 11:40:15.495865 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.495849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 11:40:15.496611 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.496086    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cm68k\""
Jul 07 11:40:15.496611 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.496114    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lf7bl\""
Jul 07 11:40:15.496611 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.496356    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 11:40:15.496611 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.496573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.496611 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.496600    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 11:40:15.498251 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.498233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.499681 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.499663    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.501025 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.501009    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:15.502020 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.502002    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 11:40:15.502020 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.502017    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:15.502551 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.502537    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:15.502639 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.502624    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:15.504021 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.504008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.504074 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.504053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:15.504548 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.504522    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b6bq8\""
Jul 07 11:40:15.504708 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.504694    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 11:40:15.504773 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.504738    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 11:40:15.504828 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.504695    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 11:40:15.504892 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.504878    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 11:40:15.504967 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.504951    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 11:40:15.505010 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505001    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 11:40:15.505230 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505201    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 11:40:15.505309 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.505544 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505530    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 11:40:15.505605 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505596    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 11:40:15.505670 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505655    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 11:40:15.505721 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505702    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 11:40:15.505721 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505658    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 11:40:15.505818 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505657    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tjp8c\""
Jul 07 11:40:15.505818 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505707    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zksct\""
Jul 07 11:40:15.505818 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x7b9p\""
Jul 07 11:40:15.505907 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505822    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xmn67\""
Jul 07 11:40:15.505995 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505982    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 11:40:15.506032 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.505985    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 11:40:15.506771 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.506755    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 11:40:15.506998 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.506983    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 11:40:15.508084 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d4f48aa-03cb-46c7-8b68-210415f6f09f-iptables-alerter-script\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.508140 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d4f48aa-03cb-46c7-8b68-210415f6f09f-host-slash\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.508140 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-var-lib-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508140 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-kubelet\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508321 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82szr\" (UniqueName: \"kubernetes.io/projected/ec46ad3a-d11e-416b-a447-c6625fe36c16-kube-api-access-82szr\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508321 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-socket-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.508321 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovnkube-config\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508321 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovn-node-metrics-cert\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508321 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-multus-certs\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508321 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7df9n\" (UniqueName: \"kubernetes.io/projected/4d4f48aa-03cb-46c7-8b68-210415f6f09f-kube-api-access-7df9n\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-system-cni-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-tmp-dir\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-serviceca\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7294718e-ce3b-429c-a703-826faaa72cf6-konnectivity-ca\") pod \"konnectivity-agent-pd5p5\" (UID: \"7294718e-ce3b-429c-a703-826faaa72cf6\") " pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2p2l\" (UniqueName: \"kubernetes.io/projected/b5cf322a-c263-4f3f-8f85-a8bb113423b5-kube-api-access-z2p2l\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-host\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnp6b\" (UniqueName: \"kubernetes.io/projected/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-kube-api-access-jnp6b\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.508595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xfk\" (UniqueName: \"kubernetes.io/projected/cc93da87-34c7-4bd9-a16b-dc777abfe795-kube-api-access-g8xfk\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-systemd\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovnkube-script-lib\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec46ad3a-d11e-416b-a447-c6625fe36c16-cni-binary-copy\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-conf-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-etc-kubernetes\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-cni-bin\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-env-overrides\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-cnibin\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-cni-bin\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-system-cni-dir\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-cnibin\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-cni-binary-copy\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508937    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-systemd-units\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.508997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.508982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ffl5\" (UniqueName: \"kubernetes.io/projected/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-kube-api-access-9ffl5\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-os-release\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-run-netns\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-netns\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-cni-multus\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-node-log\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-daemon-config\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-kubelet\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-slash\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c996e346-c3a2-4e05-97c9-f3225a44bf73-kubelet-config\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-cni-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-socket-dir-parent\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-ovn\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.509680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-device-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ddc\" (UniqueName: \"kubernetes.io/projected/57472ab7-6f1d-4a9b-9149-bf550408b54a-kube-api-access-j9ddc\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-hostroot\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-log-socket\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7294718e-ce3b-429c-a703-826faaa72cf6-agent-certs\") pod \"konnectivity-agent-pd5p5\" (UID: \"7294718e-ce3b-429c-a703-826faaa72cf6\") " pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c996e346-c3a2-4e05-97c9-f3225a44bf73-dbus\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-os-release\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-k8s-cni-cncf-io\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-hosts-file\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-etc-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-registration-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509627    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-cni-netd\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509649    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.510164 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-sys-fs\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.510654 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.509744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94cgl\" (UniqueName: \"kubernetes.io/projected/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-kube-api-access-94cgl\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.511996 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.511962    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 11:35:15 +0000 UTC" deadline="2027-12-26 09:34:28.865876046 +0000 UTC"
Jul 07 11:40:15.511996 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.511994    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12885h54m13.353883528s"
Jul 07 11:40:15.515018 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.515003    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 11:40:15.515123 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.515108    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 11:40:15.515181 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.515162    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"
Jul 07 11:40:15.515388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.515360    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal"]
Jul 07 11:40:15.515727 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.515711    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 11:40:15.515803 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.515727    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jfcw7\""
Jul 07 11:40:15.519274 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.519244    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 11:40:15.526692 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.526678    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 11:40:15.526752 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.526676    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal"]
Jul 07 11:40:15.551935 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.551865    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hsgrp"
Jul 07 11:40:15.562602 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.562579    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hsgrp"
Jul 07 11:40:15.603254 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.603235    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 11:40:15.610208 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.610319 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-log-socket\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610319 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7294718e-ce3b-429c-a703-826faaa72cf6-agent-certs\") pod \"konnectivity-agent-pd5p5\" (UID: \"7294718e-ce3b-429c-a703-826faaa72cf6\") " pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:15.610319 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c996e346-c3a2-4e05-97c9-f3225a44bf73-dbus\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-os-release\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-k8s-cni-cncf-io\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-hosts-file\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610367    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-log-socket\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-etc-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610372    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610412    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-k8s-cni-cncf-io\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610412    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-os-release\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-registration-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610450    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-etc-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610455    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-hosts-file\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.610473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610474    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-registration-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysctl-conf\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610505    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c996e346-c3a2-4e05-97c9-f3225a44bf73-dbus\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-cni-netd\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610585    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-run-ovn-kubernetes\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610619    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-cni-netd\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-sys-fs\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94cgl\" (UniqueName: \"kubernetes.io/projected/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-kube-api-access-94cgl\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610668    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.610681    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610699    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-sys-fs\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d4f48aa-03cb-46c7-8b68-210415f6f09f-iptables-alerter-script\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610682    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.610757    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret podName:c996e346-c3a2-4e05-97c9-f3225a44bf73 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:16.110735364 +0000 UTC m=+2.000929407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret") pod "global-pull-secret-syncer-85pvm" (UID: "c996e346-c3a2-4e05-97c9-f3225a44bf73") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:15.610931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d4f48aa-03cb-46c7-8b68-210415f6f09f-host-slash\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610807    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d4f48aa-03cb-46c7-8b68-210415f6f09f-host-slash\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-var-lib-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-kubelet\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82szr\" (UniqueName: \"kubernetes.io/projected/ec46ad3a-d11e-416b-a447-c6625fe36c16-kube-api-access-82szr\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-socket-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610857    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-var-lib-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610890    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-kubelet\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovnkube-config\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.610996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovn-node-metrics-cert\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611010    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-socket-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-multus-certs\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7df9n\" (UniqueName: \"kubernetes.io/projected/4d4f48aa-03cb-46c7-8b68-210415f6f09f-kube-api-access-7df9n\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611108    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-multus-certs\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-system-cni-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-tmp-dir\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.611731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-serviceca\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7294718e-ce3b-429c-a703-826faaa72cf6-konnectivity-ca\") pod \"konnectivity-agent-pd5p5\" (UID: \"7294718e-ce3b-429c-a703-826faaa72cf6\") " pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2p2l\" (UniqueName: \"kubernetes.io/projected/b5cf322a-c263-4f3f-8f85-a8bb113423b5-kube-api-access-z2p2l\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-host\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnp6b\" (UniqueName: \"kubernetes.io/projected/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-kube-api-access-jnp6b\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysconfig\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xfk\" (UniqueName: \"kubernetes.io/projected/cc93da87-34c7-4bd9-a16b-dc777abfe795-kube-api-access-g8xfk\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-systemd\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611427    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4d4f48aa-03cb-46c7-8b68-210415f6f09f-iptables-alerter-script\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovnkube-script-lib\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec46ad3a-d11e-416b-a447-c6625fe36c16-cni-binary-copy\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-conf-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611529    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovnkube-config\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-etc-kubernetes\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-cni-bin\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.612538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611590    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-env-overrides\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-cnibin\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-cni-bin\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611672    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-serviceca\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611769    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.611848    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.611903    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs podName:6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:16.111886243 +0000 UTC m=+2.002080271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs") pod "network-metrics-daemon-lrftd" (UID: "6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611952    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-system-cni-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611987    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7294718e-ce3b-429c-a703-826faaa72cf6-konnectivity-ca\") pod \"konnectivity-agent-pd5p5\" (UID: \"7294718e-ce3b-429c-a703-826faaa72cf6\") " pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612010    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-env-overrides\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612053    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-systemd\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612199    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-tmp-dir\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.611679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612262    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-host\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-systemd\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-tuned\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.613339 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612319    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-conf-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-system-cni-dir\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612368    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-etc-kubernetes\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612408    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-cnibin\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612369    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-cnibin\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612420    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-system-cni-dir\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-cni-binary-copy\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612469    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-cni-bin\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-systemd-units\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ffl5\" (UniqueName: \"kubernetes.io/projected/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-kube-api-access-9ffl5\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612543    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovnkube-script-lib\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-modprobe-d\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-os-release\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612628    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-cni-bin\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612632    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-openvswitch\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-run-netns\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612688    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-systemd-units\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612707    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec46ad3a-d11e-416b-a447-c6625fe36c16-cni-binary-copy\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612713    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-run-netns\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612740    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc93da87-34c7-4bd9-a16b-dc777abfe795-os-release\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-netns\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612794    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-cnibin\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612877    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-cni-multus\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612918    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-run-netns\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-host-var-lib-cni-multus\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.612994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-sys\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-node-log\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613056    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-daemon-config\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613122    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-node-log\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-var-lib-kubelet\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613155    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-cni-binary-copy\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.614997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-run\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-kubelet\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-slash\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613293    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c996e346-c3a2-4e05-97c9-f3225a44bf73-kubelet-config\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-cni-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-socket-dir-parent\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-tmp\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-ovn\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613414    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-kubelet\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613460    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c996e346-c3a2-4e05-97c9-f3225a44bf73-kubelet-config\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613467    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-device-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cf322a-c263-4f3f-8f85-a8bb113423b5-device-dir\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-kubernetes\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysctl-d\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9vj9\" (UniqueName: \"kubernetes.io/projected/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-kube-api-access-x9vj9\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613575    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-cni-dir\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.615615 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ddc\" (UniqueName: \"kubernetes.io/projected/57472ab7-6f1d-4a9b-9149-bf550408b54a-kube-api-access-j9ddc\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613350    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-host-slash\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-hostroot\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613613    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-daemon-config\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613619    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57472ab7-6f1d-4a9b-9149-bf550408b54a-run-ovn\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-lib-modules\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-host\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613665    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-hostroot\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613529    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ec46ad3a-d11e-416b-a447-c6625fe36c16-multus-socket-dir-parent\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.613698    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cc93da87-34c7-4bd9-a16b-dc777abfe795-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.614724    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7294718e-ce3b-429c-a703-826faaa72cf6-agent-certs\") pod \"konnectivity-agent-pd5p5\" (UID: \"7294718e-ce3b-429c-a703-826faaa72cf6\") " pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:15.616094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.615422    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57472ab7-6f1d-4a9b-9149-bf550408b54a-ovn-node-metrics-cert\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.619496 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.619474    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7df9n\" (UniqueName: \"kubernetes.io/projected/4d4f48aa-03cb-46c7-8b68-210415f6f09f-kube-api-access-7df9n\") pod \"iptables-alerter-jb4xk\" (UID: \"4d4f48aa-03cb-46c7-8b68-210415f6f09f\") " pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:15.620304 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.620285    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94cgl\" (UniqueName: \"kubernetes.io/projected/aa3fdbfa-5535-4d08-a8f0-8052596a9a02-kube-api-access-94cgl\") pod \"node-resolver-vcqtg\" (UID: \"aa3fdbfa-5535-4d08-a8f0-8052596a9a02\") " pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:15.621474 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.621451    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:15.621566 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.621479    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:15.621566 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.621493    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6l2vw for pod openshift-network-diagnostics/network-check-target-82drw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:15.621566 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.621459    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82szr\" (UniqueName: \"kubernetes.io/projected/ec46ad3a-d11e-416b-a447-c6625fe36c16-kube-api-access-82szr\") pod \"multus-697hj\" (UID: \"ec46ad3a-d11e-416b-a447-c6625fe36c16\") " pod="openshift-multus/multus-697hj"
Jul 07 11:40:15.621727 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:15.621578    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw podName:074e34fb-c8fc-4531-a2f2-4978bdff12d0 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:16.121560129 +0000 UTC m=+2.011754168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6l2vw" (UniqueName: "kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw") pod "network-check-target-82drw" (UID: "074e34fb-c8fc-4531-a2f2-4978bdff12d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:15.623391 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.623362    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2p2l\" (UniqueName: \"kubernetes.io/projected/b5cf322a-c263-4f3f-8f85-a8bb113423b5-kube-api-access-z2p2l\") pod \"aws-ebs-csi-driver-node-hlmfd\" (UID: \"b5cf322a-c263-4f3f-8f85-a8bb113423b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:15.623523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.623502    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnp6b\" (UniqueName: \"kubernetes.io/projected/f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4-kube-api-access-jnp6b\") pod \"node-ca-5h8h7\" (UID: \"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4\") " pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:15.624120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.624101    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ffl5\" (UniqueName: \"kubernetes.io/projected/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-kube-api-access-9ffl5\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:15.624342 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.624321    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ddc\" (UniqueName: \"kubernetes.io/projected/57472ab7-6f1d-4a9b-9149-bf550408b54a-kube-api-access-j9ddc\") pod \"ovnkube-node-l5kdk\" (UID: \"57472ab7-6f1d-4a9b-9149-bf550408b54a\") " pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:15.624401 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.624384    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xfk\" (UniqueName: \"kubernetes.io/projected/cc93da87-34c7-4bd9-a16b-dc777abfe795-kube-api-access-g8xfk\") pod \"multus-additional-cni-plugins-bv5rp\" (UID: \"cc93da87-34c7-4bd9-a16b-dc777abfe795\") " pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:15.631579 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:15.631563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jb4xk"
Jul 07 11:40:16.841658 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.841638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:16.846191 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.846175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:16.886030 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.886007    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:16.887973 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:16.887942    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32639e909a677f36a8e629fed59bab2.slice/crio-a1af7ede01a962b4c9d1a5a7d7eba4f3f71e71886dd51bf84d3fdc704a34d909 WatchSource:0}: Error finding container a1af7ede01a962b4c9d1a5a7d7eba4f3f71e71886dd51bf84d3fdc704a34d909: Status 404 returned error can't find the container with id a1af7ede01a962b4c9d1a5a7d7eba4f3f71e71886dd51bf84d3fdc704a34d909
Jul 07 11:40:16.888309 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:16.888288    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90e1b7edb9e4ce00addd9710b3030b5.slice/crio-3772e987a648f0560e2e215168e87e8d8a260829068cb79ec90760944c7b0096 WatchSource:0}: Error finding container 3772e987a648f0560e2e215168e87e8d8a260829068cb79ec90760944c7b0096: Status 404 returned error can't find the container with id 3772e987a648f0560e2e215168e87e8d8a260829068cb79ec90760944c7b0096
Jul 07 11:40:16.893223 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.893209    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:40:16.903214 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysconfig\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903340 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903320    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysconfig\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-systemd\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-tuned\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-modprobe-d\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-sys\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-var-lib-kubelet\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-run\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903653    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-modprobe-d\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.903680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903544    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-systemd\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-tmp\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903717    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-sys\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-kubernetes\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903777    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-run\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysctl-d\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903817    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-var-lib-kubelet\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903843    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-kubernetes\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9vj9\" (UniqueName: \"kubernetes.io/projected/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-kube-api-access-x9vj9\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-lib-modules\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-host\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903959    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysctl-d\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.903997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysctl-conf\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.904006    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-host\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.904028    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-lib-modules\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.904637 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.904189    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-sysctl-conf\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.906010 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.905991    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-etc-tuned\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.906092 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.906035    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-tmp\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:16.911837 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:16.911816    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9vj9\" (UniqueName: \"kubernetes.io/projected/d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947-kube-api-access-x9vj9\") pod \"tuned-sbkwj\" (UID: \"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947\") " pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:17.012198 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.012125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-697hj"
Jul 07 11:40:17.020153 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.020125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec46ad3a_d11e_416b_a447_c6625fe36c16.slice/crio-c64235ba37ceb2cef111065f1afd2056b3ed90a715f8e924d183ce7fb097e631 WatchSource:0}: Error finding container c64235ba37ceb2cef111065f1afd2056b3ed90a715f8e924d183ce7fb097e631: Status 404 returned error can't find the container with id c64235ba37ceb2cef111065f1afd2056b3ed90a715f8e924d183ce7fb097e631
Jul 07 11:40:17.041396 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.041372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd"
Jul 07 11:40:17.048997 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.048973    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cf322a_c263_4f3f_8f85_a8bb113423b5.slice/crio-340b5f960e31e443f4435af347df9c55d0d83ec7b461a6ffbfe9cb166a79abf9 WatchSource:0}: Error finding container 340b5f960e31e443f4435af347df9c55d0d83ec7b461a6ffbfe9cb166a79abf9: Status 404 returned error can't find the container with id 340b5f960e31e443f4435af347df9c55d0d83ec7b461a6ffbfe9cb166a79abf9
Jul 07 11:40:17.071456 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.071417    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcqtg"
Jul 07 11:40:17.078038 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.078007    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3fdbfa_5535_4d08_a8f0_8052596a9a02.slice/crio-acf7c6f6617b1750d2295d7dfae87c0ef285956a05963a979b85e3806c0c97de WatchSource:0}: Error finding container acf7c6f6617b1750d2295d7dfae87c0ef285956a05963a979b85e3806c0c97de: Status 404 returned error can't find the container with id acf7c6f6617b1750d2295d7dfae87c0ef285956a05963a979b85e3806c0c97de
Jul 07 11:40:17.084192 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.084175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5h8h7"
Jul 07 11:40:17.090613 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.090590    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a1c866_f2cb_4c8d_805a_9ed813d9cbc4.slice/crio-7023faed74929c907bd230599a5e2e0c10b3490fa2dd0e5ee992b5931835b728 WatchSource:0}: Error finding container 7023faed74929c907bd230599a5e2e0c10b3490fa2dd0e5ee992b5931835b728: Status 404 returned error can't find the container with id 7023faed74929c907bd230599a5e2e0c10b3490fa2dd0e5ee992b5931835b728
Jul 07 11:40:17.104637 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.104612    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bv5rp"
Jul 07 11:40:17.111673 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.111652    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc93da87_34c7_4bd9_a16b_dc777abfe795.slice/crio-a84b79b4c53424c8e3d4642a04c893751dd627e5531c307208e12d10d64c20d5 WatchSource:0}: Error finding container a84b79b4c53424c8e3d4642a04c893751dd627e5531c307208e12d10d64c20d5: Status 404 returned error can't find the container with id a84b79b4c53424c8e3d4642a04c893751dd627e5531c307208e12d10d64c20d5
Jul 07 11:40:17.166358 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.166324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sbkwj"
Jul 07 11:40:17.173931 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.173899    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bd7fa7_d91d_4cd3_8db8_2e2d3df39947.slice/crio-5f10e66adaed39f5010aff3d9065dba057186d57b065c1dda2082c1bd274e62d WatchSource:0}: Error finding container 5f10e66adaed39f5010aff3d9065dba057186d57b065c1dda2082c1bd274e62d: Status 404 returned error can't find the container with id 5f10e66adaed39f5010aff3d9065dba057186d57b065c1dda2082c1bd274e62d
Jul 07 11:40:17.193432 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.193410    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4f48aa_03cb_46c7_8b68_210415f6f09f.slice/crio-2b89055c40a376097d19fefcb663463443623089d14301d91233f05fc59dafc3 WatchSource:0}: Error finding container 2b89055c40a376097d19fefcb663463443623089d14301d91233f05fc59dafc3: Status 404 returned error can't find the container with id 2b89055c40a376097d19fefcb663463443623089d14301d91233f05fc59dafc3
Jul 07 11:40:17.306891 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.306854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:17.307047 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.306918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:17.307047 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.307015    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:17.307047 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.307037    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:17.307199 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.307090    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs podName:6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:18.307071983 +0000 UTC m=+3.008004840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs") pod "network-metrics-daemon-lrftd" (UID: "6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:17.307199 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.307111    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret podName:c996e346-c3a2-4e05-97c9-f3225a44bf73 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:18.307101963 +0000 UTC m=+3.008034803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret") pod "global-pull-secret-syncer-85pvm" (UID: "c996e346-c3a2-4e05-97c9-f3225a44bf73") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:17.307703 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.307674    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57472ab7_6f1d_4a9b_9149_bf550408b54a.slice/crio-58dabf7c01f04a7997c0edb1f045dc9c59b67d30567e369309dc6265c94a83e4 WatchSource:0}: Error finding container 58dabf7c01f04a7997c0edb1f045dc9c59b67d30567e369309dc6265c94a83e4: Status 404 returned error can't find the container with id 58dabf7c01f04a7997c0edb1f045dc9c59b67d30567e369309dc6265c94a83e4
Jul 07 11:40:17.407579 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.407537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:17.407736 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.407701    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:17.407736 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.407720    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:17.407853 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.407759    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6l2vw for pod openshift-network-diagnostics/network-check-target-82drw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:17.407853 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.407824    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw podName:074e34fb-c8fc-4531-a2f2-4978bdff12d0 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:18.407805288 +0000 UTC m=+3.108738130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6l2vw" (UniqueName: "kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw") pod "network-check-target-82drw" (UID: "074e34fb-c8fc-4531-a2f2-4978bdff12d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:17.430516 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:17.430459    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7294718e_ce3b_429c_a703_826faaa72cf6.slice/crio-4a0786424f78d86bd715d21d972035e3b1aec89ad76d71bacc3bfa82c77dab5a WatchSource:0}: Error finding container 4a0786424f78d86bd715d21d972035e3b1aec89ad76d71bacc3bfa82c77dab5a: Status 404 returned error can't find the container with id 4a0786424f78d86bd715d21d972035e3b1aec89ad76d71bacc3bfa82c77dab5a
Jul 07 11:40:17.753718 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.753567    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 11:35:16 +0000 UTC" deadline="2028-01-16 21:20:08.851426883 +0000 UTC"
Jul 07 11:40:17.753718 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.753626    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13401h39m51.09780555s"
Jul 07 11:40:17.882641 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.882051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:17.882641 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:17.882188    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:17.896830 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.896770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerStarted","Data":"a84b79b4c53424c8e3d4642a04c893751dd627e5531c307208e12d10d64c20d5"}
Jul 07 11:40:17.913258 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.913184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-697hj" event={"ID":"ec46ad3a-d11e-416b-a447-c6625fe36c16","Type":"ContainerStarted","Data":"c64235ba37ceb2cef111065f1afd2056b3ed90a715f8e924d183ce7fb097e631"}
Jul 07 11:40:17.915603 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.915550    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 11:40:17.918875 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.918843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal" event={"ID":"e90e1b7edb9e4ce00addd9710b3030b5","Type":"ContainerStarted","Data":"3772e987a648f0560e2e215168e87e8d8a260829068cb79ec90760944c7b0096"}
Jul 07 11:40:17.927998 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.927937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jb4xk" event={"ID":"4d4f48aa-03cb-46c7-8b68-210415f6f09f","Type":"ContainerStarted","Data":"2b89055c40a376097d19fefcb663463443623089d14301d91233f05fc59dafc3"}
Jul 07 11:40:17.933008 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.932937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sbkwj" event={"ID":"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947","Type":"ContainerStarted","Data":"5f10e66adaed39f5010aff3d9065dba057186d57b065c1dda2082c1bd274e62d"}
Jul 07 11:40:17.949562 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.949526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5h8h7" event={"ID":"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4","Type":"ContainerStarted","Data":"7023faed74929c907bd230599a5e2e0c10b3490fa2dd0e5ee992b5931835b728"}
Jul 07 11:40:17.964778 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.964711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcqtg" event={"ID":"aa3fdbfa-5535-4d08-a8f0-8052596a9a02","Type":"ContainerStarted","Data":"acf7c6f6617b1750d2295d7dfae87c0ef285956a05963a979b85e3806c0c97de"}
Jul 07 11:40:17.978039 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.977999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd" event={"ID":"b5cf322a-c263-4f3f-8f85-a8bb113423b5","Type":"ContainerStarted","Data":"340b5f960e31e443f4435af347df9c55d0d83ec7b461a6ffbfe9cb166a79abf9"}
Jul 07 11:40:17.989188 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:17.989000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" event={"ID":"b32639e909a677f36a8e629fed59bab2","Type":"ContainerStarted","Data":"a1af7ede01a962b4c9d1a5a7d7eba4f3f71e71886dd51bf84d3fdc704a34d909"}
Jul 07 11:40:18.001830 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.001796    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pd5p5" event={"ID":"7294718e-ce3b-429c-a703-826faaa72cf6","Type":"ContainerStarted","Data":"4a0786424f78d86bd715d21d972035e3b1aec89ad76d71bacc3bfa82c77dab5a"}
Jul 07 11:40:18.012174 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.012079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"58dabf7c01f04a7997c0edb1f045dc9c59b67d30567e369309dc6265c94a83e4"}
Jul 07 11:40:18.315469 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.315374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:18.315642 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.315510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:18.315642 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.315636    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:18.315750 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.315698    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret podName:c996e346-c3a2-4e05-97c9-f3225a44bf73 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:20.31568048 +0000 UTC m=+5.016613320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret") pod "global-pull-secret-syncer-85pvm" (UID: "c996e346-c3a2-4e05-97c9-f3225a44bf73") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:18.316204 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.316104    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:18.316204 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.316158    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs podName:6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:20.316141103 +0000 UTC m=+5.017073943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs") pod "network-metrics-daemon-lrftd" (UID: "6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:18.416747 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.416704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:18.416936 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.416880    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:18.416936 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.416899    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:18.416936 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.416912    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6l2vw for pod openshift-network-diagnostics/network-check-target-82drw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:18.417114 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.416972    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw podName:074e34fb-c8fc-4531-a2f2-4978bdff12d0 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:20.416952281 +0000 UTC m=+5.117885138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6l2vw" (UniqueName: "kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw") pod "network-check-target-82drw" (UID: "074e34fb-c8fc-4531-a2f2-4978bdff12d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:18.754074 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.754030    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 11:35:16 +0000 UTC" deadline="2028-02-22 01:50:49.909186853 +0000 UTC"
Jul 07 11:40:18.754074 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.754076    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14270h10m31.155119016s"
Jul 07 11:40:18.881035 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.880225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:18.881035 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.880363    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:18.881035 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:18.880788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:18.881035 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:18.880885    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:19.880836 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:19.880244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:19.880836 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:19.880471    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:20.333240 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:20.333153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:20.333240 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:20.333217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:20.333454 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.333395    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:20.333532 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.333459    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs podName:6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:24.333440046 +0000 UTC m=+9.034372885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs") pod "network-metrics-daemon-lrftd" (UID: "6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:20.333937 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.333920    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:20.334013 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.333975    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret podName:c996e346-c3a2-4e05-97c9-f3225a44bf73 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:24.333958967 +0000 UTC m=+9.034891819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret") pod "global-pull-secret-syncer-85pvm" (UID: "c996e346-c3a2-4e05-97c9-f3225a44bf73") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:20.433705 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:20.433671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:20.433891 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.433835    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:20.433891 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.433855    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:20.433891 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.433868    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6l2vw for pod openshift-network-diagnostics/network-check-target-82drw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:20.434057 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.433931    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw podName:074e34fb-c8fc-4531-a2f2-4978bdff12d0 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:24.433912353 +0000 UTC m=+9.134845193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6l2vw" (UniqueName: "kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw") pod "network-check-target-82drw" (UID: "074e34fb-c8fc-4531-a2f2-4978bdff12d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:20.879489 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:20.879443    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:20.879801 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:20.879783    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:20.879917 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.879891    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:20.880373 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:20.880341    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:21.880613 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:21.880088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:21.880613 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:21.880238    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:22.880139 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:22.880103    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:22.880299 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:22.880249    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:22.880386 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:22.880365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:22.880475 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:22.880454    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:23.880880 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:23.880583    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:23.880880 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:23.880738    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:24.368946 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:24.368907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:24.369128 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:24.368962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:24.369128 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.369083    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:24.369239 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.369128    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:24.369239 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.369156    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret podName:c996e346-c3a2-4e05-97c9-f3225a44bf73 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:32.369135702 +0000 UTC m=+17.070068543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret") pod "global-pull-secret-syncer-85pvm" (UID: "c996e346-c3a2-4e05-97c9-f3225a44bf73") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:24.369239 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.369183    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs podName:6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:32.369166561 +0000 UTC m=+17.070099403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs") pod "network-metrics-daemon-lrftd" (UID: "6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:24.469903 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:24.469866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:24.470084 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.470044    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:24.470084 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.470063    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:24.470084 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.470077    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6l2vw for pod openshift-network-diagnostics/network-check-target-82drw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:24.470252 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.470134    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw podName:074e34fb-c8fc-4531-a2f2-4978bdff12d0 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:32.470116554 +0000 UTC m=+17.171049395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6l2vw" (UniqueName: "kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw") pod "network-check-target-82drw" (UID: "074e34fb-c8fc-4531-a2f2-4978bdff12d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:24.879730 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:24.879411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:24.879730 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:24.879453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:24.879730 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.879579    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:24.879730 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:24.879664    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:25.881424 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:25.881310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:25.881424 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:25.881413    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:26.879776 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:26.879740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:26.879943 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:26.879881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:26.880010 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:26.879970    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:26.880010 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:26.879875    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:27.880247 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:27.880213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:27.880708 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:27.880332    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:28.879873 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:28.879835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:28.880049 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:28.879965    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:28.880049 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:28.880028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:28.880163 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:28.880143    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:29.879659 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:29.879617    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:29.880065 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:29.879757    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:30.880016 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:30.879979    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:30.880439 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:30.879980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:30.880439 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:30.880097    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:30.880439 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:30.880162    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:31.880287 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:31.880256    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:31.880749 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:31.880396    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:32.429002 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:32.428965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:32.429218 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:32.429017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:32.429218 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.429133    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:32.429218 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.429165    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:32.429218 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.429215    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret podName:c996e346-c3a2-4e05-97c9-f3225a44bf73 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:48.42919281 +0000 UTC m=+33.130125647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret") pod "global-pull-secret-syncer-85pvm" (UID: "c996e346-c3a2-4e05-97c9-f3225a44bf73") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:32.429436 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.429237    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs podName:6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:48.429225067 +0000 UTC m=+33.130157904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs") pod "network-metrics-daemon-lrftd" (UID: "6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:32.529701 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:32.529659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:32.529911 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.529856    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:32.529911 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.529879    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:32.529911 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.529888    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6l2vw for pod openshift-network-diagnostics/network-check-target-82drw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:32.530035 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.529949    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw podName:074e34fb-c8fc-4531-a2f2-4978bdff12d0 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:48.529932431 +0000 UTC m=+33.230865274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6l2vw" (UniqueName: "kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw") pod "network-check-target-82drw" (UID: "074e34fb-c8fc-4531-a2f2-4978bdff12d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:32.879560 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:32.879520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:32.879762 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:32.879536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:32.879762 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.879675    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:32.879888 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:32.879758    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:33.879742 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:33.879705    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:33.880293 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:33.879845    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:34.880196 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:34.880154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:34.880643 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:34.880156    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:34.880643 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:34.880291    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:34.880643 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:34.880354    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:35.879997 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:35.879958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:35.880187 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:35.880064    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:36.879625 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:36.879394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:36.879949 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:36.879444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:36.879949 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:36.879715    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:36.879949 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:36.879782    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:37.073193 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.073105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal" event={"ID":"e90e1b7edb9e4ce00addd9710b3030b5","Type":"ContainerStarted","Data":"5f9a847ae3617518f533272a6dcbc272b83f8a78e11cc70c2ecc1311473b3e91"}
Jul 07 11:40:37.074740 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.074707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sbkwj" event={"ID":"d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947","Type":"ContainerStarted","Data":"4908600a0e68d2dc727bc541f0bca7b08c1a6856ab99f8fb13f8a3c9ca2a3af3"}
Jul 07 11:40:37.076912 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.076885    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:40:37.077533 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.077329    2565 generic.go:356] "Generic (PLEG): container finished" podID="57472ab7-6f1d-4a9b-9149-bf550408b54a" containerID="52d9945c2e21e95a649c60b3996621ede27ca2d3eef467e29f6a7d037dfd79a9" exitCode=1
Jul 07 11:40:37.077533 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.077369    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerDied","Data":"52d9945c2e21e95a649c60b3996621ede27ca2d3eef467e29f6a7d037dfd79a9"}
Jul 07 11:40:37.077533 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.077443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"387a2775dda3faf6b61e39ae624b26b822c4af8e735aa4acb251066801eabb9d"}
Jul 07 11:40:37.101097 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.101015    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-227.ec2.internal" podStartSLOduration=21.100993038 podStartE2EDuration="21.100993038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:37.100921699 +0000 UTC m=+21.801854556" watchObservedRunningTime="2026-07-07 11:40:37.100993038 +0000 UTC m=+21.801925897"
Jul 07 11:40:37.879737 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:37.879574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:37.880255 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:37.879820    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:38.081058 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.080992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd" event={"ID":"b5cf322a-c263-4f3f-8f85-a8bb113423b5","Type":"ContainerStarted","Data":"f8f0dc7f984497ef821cbe3ae29d6988f55a286e37f117cb57fba753db4b0656"}
Jul 07 11:40:38.082307 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.082256    2565 generic.go:356] "Generic (PLEG): container finished" podID="b32639e909a677f36a8e629fed59bab2" containerID="8fc21a149b4f8409a62a3e0c9fe47015319f667bb9fe605c061d5fc35caee741" exitCode=0
Jul 07 11:40:38.082377 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.082336    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" event={"ID":"b32639e909a677f36a8e629fed59bab2","Type":"ContainerDied","Data":"8fc21a149b4f8409a62a3e0c9fe47015319f667bb9fe605c061d5fc35caee741"}
Jul 07 11:40:38.083881 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.083858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pd5p5" event={"ID":"7294718e-ce3b-429c-a703-826faaa72cf6","Type":"ContainerStarted","Data":"f982a4dac5aeb50d99dc4cf1c6c4a0d65d60f520a29b07118942e5715e96ecac"}
Jul 07 11:40:38.086458 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.086441    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:40:38.086824 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.086806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"6c2478b6bb39e8c275688ac2ede0ee5c31deadbf8a1e1e33c4fc37bcfb5f543b"}
Jul 07 11:40:38.086897 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.086866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"e5171d481c650d3d612458041fb3ca385e15ff5cbfc5c023448f5158a589a49c"}
Jul 07 11:40:38.086940 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.086913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"29fe3b3ba43e06bc9287b1b944fb1148d1685e7a42ecafd71c344a6162935bff"}
Jul 07 11:40:38.086981 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.086944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"3f8e11b78c11450aeceb50603ee129b1aabb9e8aeef3018d98d85e85fa355efa"}
Jul 07 11:40:38.088074 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.088057    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc93da87-34c7-4bd9-a16b-dc777abfe795" containerID="4e332ee929fa0013f43630f2187cd44932f845690f72c31a4fb86d423e5b785d" exitCode=0
Jul 07 11:40:38.088138 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.088117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerDied","Data":"4e332ee929fa0013f43630f2187cd44932f845690f72c31a4fb86d423e5b785d"}
Jul 07 11:40:38.089581 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.089557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-697hj" event={"ID":"ec46ad3a-d11e-416b-a447-c6625fe36c16","Type":"ContainerStarted","Data":"7fb67acd4d8f4b5d3b343f21c27da54be0127ca1e60dc3aa2883a2afc3d5588c"}
Jul 07 11:40:38.092173 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.092147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5h8h7" event={"ID":"f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4","Type":"ContainerStarted","Data":"07e325f4d14ee0a4a1c4dda3bdbd10be7b795a279fc5f03cb172a8af38155981"}
Jul 07 11:40:38.093453 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.093432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcqtg" event={"ID":"aa3fdbfa-5535-4d08-a8f0-8052596a9a02","Type":"ContainerStarted","Data":"586894a05895107033229ada808415566a1ad709d205ac7a6dc1fbba5f160be7"}
Jul 07 11:40:38.103202 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.103159    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sbkwj" podStartSLOduration=2.614574522 podStartE2EDuration="22.103144882s" totalImagesPullingTime="19.48857036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:37.12299154 +0000 UTC m=+21.823924399" watchObservedRunningTime="2026-07-07 11:40:38.103144882 +0000 UTC m=+22.804077740"
Jul 07 11:40:38.124600 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.124336    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-697hj" podStartSLOduration=3.143536198 podStartE2EDuration="23.124316252s" totalImagesPullingTime="19.980780054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:38.123845823 +0000 UTC m=+22.824778682" watchObservedRunningTime="2026-07-07 11:40:38.124316252 +0000 UTC m=+22.825249112"
Jul 07 11:40:38.139568 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.139509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vcqtg" podStartSLOduration=3.584612149 podStartE2EDuration="23.139490897s" totalImagesPullingTime="19.554878748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:38.139294534 +0000 UTC m=+22.840227394" watchObservedRunningTime="2026-07-07 11:40:38.139490897 +0000 UTC m=+22.840423751"
Jul 07 11:40:38.207640 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.207582    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5h8h7" podStartSLOduration=3.637806231 podStartE2EDuration="23.207558926s" totalImagesPullingTime="19.569752695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:38.207305276 +0000 UTC m=+22.908238136" watchObservedRunningTime="2026-07-07 11:40:38.207558926 +0000 UTC m=+22.908491786"
Jul 07 11:40:38.208236 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.208197    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pd5p5" podStartSLOduration=4.007476816 podStartE2EDuration="23.208183391s" totalImagesPullingTime="19.200706575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:38.181544314 +0000 UTC m=+22.882477176" watchObservedRunningTime="2026-07-07 11:40:38.208183391 +0000 UTC m=+22.909116300"
Jul 07 11:40:38.879496 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.879451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:38.879668 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:38.879448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:38.879668 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:38.879603    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:38.879668 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:38.879635    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:39.098200 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.098162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd" event={"ID":"b5cf322a-c263-4f3f-8f85-a8bb113423b5","Type":"ContainerStarted","Data":"8514239e10b8c44477b6c6cbaaeb417fc65b53567ee825b40be2f988445312fc"}
Jul 07 11:40:39.100501 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.100400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" event={"ID":"b32639e909a677f36a8e629fed59bab2","Type":"ContainerStarted","Data":"6bd69b5c2b8d97183d61988c820f93524ad66250196f0f91953071bd7a747b6a"}
Jul 07 11:40:39.102940 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.102027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jb4xk" event={"ID":"4d4f48aa-03cb-46c7-8b68-210415f6f09f","Type":"ContainerStarted","Data":"5a867ac196557422be3d08f14d3dd5840af8d3a0edb84765501f8defb3cc59a2"}
Jul 07 11:40:39.105135 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.105113    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 11:40:39.137097 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.136994    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jb4xk" podStartSLOduration=4.6701314929999995 podStartE2EDuration="24.1369733s" totalImagesPullingTime="19.466841807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:39.136655182 +0000 UTC m=+23.837588041" watchObservedRunningTime="2026-07-07 11:40:39.1369733 +0000 UTC m=+23.837906160"
Jul 07 11:40:39.137745 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.137706    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-227.ec2.internal" podStartSLOduration=23.137690524 podStartE2EDuration="23.137690524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:39.119352685 +0000 UTC m=+23.820285545" watchObservedRunningTime="2026-07-07 11:40:39.137690524 +0000 UTC m=+23.838623384"
Jul 07 11:40:39.764133 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.763895    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T11:40:39.105130235Z","UUID":"d4276ec2-6aae-41be-a44e-5a4278daa287","Handler":null,"Name":"","Endpoint":""}
Jul 07 11:40:39.765903 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.765881    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 11:40:39.766041 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.765926    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 11:40:39.880388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:39.880354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:39.880869 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:39.880529    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:40.107415 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:40.107386    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:40:40.108036 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:40.108010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"7513043509a2b06964ab30f2a1d7c354b93741f4395d93314032bf7f56dca40f"}
Jul 07 11:40:40.879689 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:40.879649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:40.879689 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:40.879683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:40.879953 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:40.879777    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:40.879953 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:40.879892    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:41.112354 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:41.112312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd" event={"ID":"b5cf322a-c263-4f3f-8f85-a8bb113423b5","Type":"ContainerStarted","Data":"84eb5b070c09829c42f124337668edf7ec7167afff31f71846ef0c72de0011f4"}
Jul 07 11:40:41.139971 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:41.139878    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hlmfd" podStartSLOduration=4.279987812 podStartE2EDuration="26.139861223s" totalImagesPullingTime="21.859873411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:41.139242711 +0000 UTC m=+25.840175562" watchObservedRunningTime="2026-07-07 11:40:41.139861223 +0000 UTC m=+25.840794112"
Jul 07 11:40:41.865819 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:41.865775    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:41.866430 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:41.866398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:41.880306 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:41.880276    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:41.880448 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:41.880382    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:42.119696 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:42.119575    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:40:42.120820 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:42.120095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"f4ab8fca39f254d1e9d777cbfc3970ce2a2ac9c1c79105401729d2746eb9d0bd"}
Jul 07 11:40:42.120820 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:42.120466    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:42.121368 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:42.120829    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pd5p5"
Jul 07 11:40:42.121368 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:42.120858    2565 scope.go:117] "RemoveContainer" containerID="52d9945c2e21e95a649c60b3996621ede27ca2d3eef467e29f6a7d037dfd79a9"
Jul 07 11:40:42.880523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:42.880313    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:42.880688 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:42.880371    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:42.880688 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:42.880600    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:42.880688 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:42.880662    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:43.126265 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.126236    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:40:43.127034 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.126596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" event={"ID":"57472ab7-6f1d-4a9b-9149-bf550408b54a","Type":"ContainerStarted","Data":"b4b4c0ff221129c2e506a3f51260b72fa6556e65adc356d9f739c29bf031c02e"}
Jul 07 11:40:43.127034 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.126898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:43.127034 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.126963    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:43.127034 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.127011    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:43.128536 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.128510    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc93da87-34c7-4bd9-a16b-dc777abfe795" containerID="d6d89bb75ee054762b25eef509d468fd5161a2ce135347f7fc048dbef7a0ddce" exitCode=0
Jul 07 11:40:43.128627 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.128591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerDied","Data":"d6d89bb75ee054762b25eef509d468fd5161a2ce135347f7fc048dbef7a0ddce"}
Jul 07 11:40:43.143659 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.143616    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:43.145394 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.144871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:40:43.160990 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.160947    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk" podStartSLOduration=8.764571337 podStartE2EDuration="28.160932389s" totalImagesPullingTime="19.396361052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:43.159802684 +0000 UTC m=+27.860735543" watchObservedRunningTime="2026-07-07 11:40:43.160932389 +0000 UTC m=+27.861865247"
Jul 07 11:40:43.880274 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:43.880239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:43.880495 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:43.880351    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:44.434738 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:44.434462    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-85pvm"]
Jul 07 11:40:44.435201 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:44.434768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:44.435201 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:44.434860    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:44.437016 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:44.436988    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lrftd"]
Jul 07 11:40:44.437162 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:44.437076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:44.437243 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:44.437167    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:44.438622 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:44.438601    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82drw"]
Jul 07 11:40:44.438704 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:44.438683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:44.438767 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:44.438750    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:45.135308 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:45.135276    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc93da87-34c7-4bd9-a16b-dc777abfe795" containerID="d85133b97ef2518d8dd6e1b637b47a3670bba641546518ac75be9bb27c4031f4" exitCode=0
Jul 07 11:40:45.135523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:45.135366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerDied","Data":"d85133b97ef2518d8dd6e1b637b47a3670bba641546518ac75be9bb27c4031f4"}
Jul 07 11:40:45.882759 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:45.882724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:45.882759 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:45.882754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:45.883201 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:45.883180    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:45.883244 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:45.883207    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:45.883283 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:45.883262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:45.883382 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:45.883361    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:47.143344 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:47.143308    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc93da87-34c7-4bd9-a16b-dc777abfe795" containerID="8b8ad77a31ebd9c985967ac2f52746a78ac8592bcf007bc3d0bc5315f7524569" exitCode=0
Jul 07 11:40:47.143797 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:47.143355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerDied","Data":"8b8ad77a31ebd9c985967ac2f52746a78ac8592bcf007bc3d0bc5315f7524569"}
Jul 07 11:40:47.879813 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:47.879777    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:47.879813 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:47.879813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:47.880122 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:47.879782    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:47.880122 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:47.879908    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-82drw" podUID="074e34fb-c8fc-4531-a2f2-4978bdff12d0"
Jul 07 11:40:47.880122 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:47.879986    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lrftd" podUID="6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2"
Jul 07 11:40:47.880122 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:47.880062    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-85pvm" podUID="c996e346-c3a2-4e05-97c9-f3225a44bf73"
Jul 07 11:40:48.440372 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:48.440306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:48.440840 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:48.440393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:48.440840 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.440474    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:48.440840 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.440554    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:48.440840 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.440574    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret podName:c996e346-c3a2-4e05-97c9-f3225a44bf73 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:20.440553739 +0000 UTC m=+65.141486581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret") pod "global-pull-secret-syncer-85pvm" (UID: "c996e346-c3a2-4e05-97c9-f3225a44bf73") : object "kube-system"/"original-pull-secret" not registered
Jul 07 11:40:48.440840 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.440611    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs podName:6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:20.440593687 +0000 UTC m=+65.141526526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs") pod "network-metrics-daemon-lrftd" (UID: "6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 11:40:48.541059 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:48.541026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:48.541228 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.541182    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 11:40:48.541228 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.541202    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 11:40:48.541228 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.541214    2565 projected.go:196] Error preparing data for projected volume kube-api-access-6l2vw for pod openshift-network-diagnostics/network-check-target-82drw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:48.541356 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:48.541268    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw podName:074e34fb-c8fc-4531-a2f2-4978bdff12d0 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:20.54125356 +0000 UTC m=+65.242186397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6l2vw" (UniqueName: "kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw") pod "network-check-target-82drw" (UID: "074e34fb-c8fc-4531-a2f2-4978bdff12d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 11:40:49.645223 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.645190    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-227.ec2.internal" event="NodeReady"
Jul 07 11:40:49.645667 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.645337    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 11:40:49.720659 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.720579    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l5mpr"]
Jul 07 11:40:49.745912 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.745875    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x6bk7"]
Jul 07 11:40:49.746388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.746348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.749283 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.749261    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4khfw\""
Jul 07 11:40:49.749411 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.749313    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 11:40:49.750038 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.750017    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 11:40:49.759766 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.759745    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l5mpr"]
Jul 07 11:40:49.759891 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.759801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6bk7"]
Jul 07 11:40:49.759891 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.759825    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4qx5d"]
Jul 07 11:40:49.759891 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.759842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:49.762513 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.762491    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 11:40:49.762874 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.762850    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-svb8z\""
Jul 07 11:40:49.762982 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.762962    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 11:40:49.763141 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.763124    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 11:40:49.777927 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.777899    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4qx5d"]
Jul 07 11:40:49.778080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.778002    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.781201 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.781180    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f587k\""
Jul 07 11:40:49.781497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.781461    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 11:40:49.781607 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.781533    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 11:40:49.782050 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.782015    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 11:40:49.782159 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.782105    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 11:40:49.849927 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.849892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9hs5\" (UniqueName: \"kubernetes.io/projected/1b7b5a55-0357-4094-b50c-528667fbf4b1-kube-api-access-p9hs5\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.850105 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.849936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/228dd7ca-ee07-481d-bd00-e8648e400bd2-data-volume\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.850105 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72vz\" (UniqueName: \"kubernetes.io/projected/8b718c89-ec23-49c6-a70a-ebe38d5b482e-kube-api-access-f72vz\") pod \"ingress-canary-x6bk7\" (UID: \"8b718c89-ec23-49c6-a70a-ebe38d5b482e\") " pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:49.850105 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27c9h\" (UniqueName: \"kubernetes.io/projected/228dd7ca-ee07-481d-bd00-e8648e400bd2-kube-api-access-27c9h\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.850105 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/228dd7ca-ee07-481d-bd00-e8648e400bd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.850277 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.850277 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/228dd7ca-ee07-481d-bd00-e8648e400bd2-crio-socket\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.850277 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850206    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b718c89-ec23-49c6-a70a-ebe38d5b482e-cert\") pod \"ingress-canary-x6bk7\" (UID: \"8b718c89-ec23-49c6-a70a-ebe38d5b482e\") " pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:49.850277 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b5a55-0357-4094-b50c-528667fbf4b1-config-volume\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.850277 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7b5a55-0357-4094-b50c-528667fbf4b1-tmp-dir\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.850463 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.850305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.880050 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.880014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:40:49.880222 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.880058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:40:49.880222 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.880014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:40:49.883645 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.883622    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 11:40:49.883845 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.883817    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vqglz\""
Jul 07 11:40:49.884153 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.884128    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 11:40:49.884357 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.884297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 11:40:49.884357 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.884301    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cprtm\""
Jul 07 11:40:49.884357 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.884347    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 11:40:49.951172 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b5a55-0357-4094-b50c-528667fbf4b1-config-volume\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.951172 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7b5a55-0357-4094-b50c-528667fbf4b1-tmp-dir\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.951388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.951388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9hs5\" (UniqueName: \"kubernetes.io/projected/1b7b5a55-0357-4094-b50c-528667fbf4b1-kube-api-access-p9hs5\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.951388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/228dd7ca-ee07-481d-bd00-e8648e400bd2-data-volume\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.951388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72vz\" (UniqueName: \"kubernetes.io/projected/8b718c89-ec23-49c6-a70a-ebe38d5b482e-kube-api-access-f72vz\") pod \"ingress-canary-x6bk7\" (UID: \"8b718c89-ec23-49c6-a70a-ebe38d5b482e\") " pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:49.951388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27c9h\" (UniqueName: \"kubernetes.io/projected/228dd7ca-ee07-481d-bd00-e8648e400bd2-kube-api-access-27c9h\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.951388 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/228dd7ca-ee07-481d-bd00-e8648e400bd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.951388 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:49.951371    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:49.951748 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951398    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.951748 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/228dd7ca-ee07-481d-bd00-e8648e400bd2-crio-socket\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.951748 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:49.951455    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls podName:228dd7ca-ee07-481d-bd00-e8648e400bd2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:50.451434461 +0000 UTC m=+35.152367318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qx5d" (UID: "228dd7ca-ee07-481d-bd00-e8648e400bd2") : secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:49.951748 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b718c89-ec23-49c6-a70a-ebe38d5b482e-cert\") pod \"ingress-canary-x6bk7\" (UID: \"8b718c89-ec23-49c6-a70a-ebe38d5b482e\") " pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:49.951748 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951599    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/228dd7ca-ee07-481d-bd00-e8648e400bd2-crio-socket\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.951748 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:49.951672    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 11:40:49.951748 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:49.951706    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls podName:1b7b5a55-0357-4094-b50c-528667fbf4b1 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:50.45169392 +0000 UTC m=+35.152626757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls") pod "dns-default-l5mpr" (UID: "1b7b5a55-0357-4094-b50c-528667fbf4b1") : secret "dns-default-metrics-tls" not found
Jul 07 11:40:49.952080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951790    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b7b5a55-0357-4094-b50c-528667fbf4b1-tmp-dir\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.952080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.951825    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1b7b5a55-0357-4094-b50c-528667fbf4b1-config-volume\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:49.952187 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.952089    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/228dd7ca-ee07-481d-bd00-e8648e400bd2-data-volume\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.952284 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.952255    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/228dd7ca-ee07-481d-bd00-e8648e400bd2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.956461 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.956341    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b718c89-ec23-49c6-a70a-ebe38d5b482e-cert\") pod \"ingress-canary-x6bk7\" (UID: \"8b718c89-ec23-49c6-a70a-ebe38d5b482e\") " pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:49.964993 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.964954    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72vz\" (UniqueName: \"kubernetes.io/projected/8b718c89-ec23-49c6-a70a-ebe38d5b482e-kube-api-access-f72vz\") pod \"ingress-canary-x6bk7\" (UID: \"8b718c89-ec23-49c6-a70a-ebe38d5b482e\") " pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:49.965523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.965498    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27c9h\" (UniqueName: \"kubernetes.io/projected/228dd7ca-ee07-481d-bd00-e8648e400bd2-kube-api-access-27c9h\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:49.965634 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:49.965549    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9hs5\" (UniqueName: \"kubernetes.io/projected/1b7b5a55-0357-4094-b50c-528667fbf4b1-kube-api-access-p9hs5\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:50.069603 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:50.069521    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x6bk7"
Jul 07 11:40:50.455905 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:50.455865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:50.456104 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:50.455960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:50.456104 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:50.456023    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:50.456104 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:50.456072    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 11:40:50.456104 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:50.456104    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls podName:228dd7ca-ee07-481d-bd00-e8648e400bd2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:51.45608344 +0000 UTC m=+36.157016283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qx5d" (UID: "228dd7ca-ee07-481d-bd00-e8648e400bd2") : secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:50.456324 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:50.456132    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls podName:1b7b5a55-0357-4094-b50c-528667fbf4b1 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:51.456115628 +0000 UTC m=+36.157048489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls") pod "dns-default-l5mpr" (UID: "1b7b5a55-0357-4094-b50c-528667fbf4b1") : secret "dns-default-metrics-tls" not found
Jul 07 11:40:51.026602 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.026559    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-4tv9c"]
Jul 07 11:40:51.028917 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.028889    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.033849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.033813    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 11:40:51.033849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.033840    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 11:40:51.034080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.033911    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 11:40:51.034080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.033840    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 11:40:51.034080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.033979    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zlmct\""
Jul 07 11:40:51.042260 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.042227    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-4tv9c"]
Jul 07 11:40:51.163914 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.163883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc99w\" (UniqueName: \"kubernetes.io/projected/966906a4-e943-4f4c-bb84-5f974dff4e85-kube-api-access-qc99w\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.164130 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.163956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/966906a4-e943-4f4c-bb84-5f974dff4e85-signing-cabundle\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.164130 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.164037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/966906a4-e943-4f4c-bb84-5f974dff4e85-signing-key\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.265454 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.265413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc99w\" (UniqueName: \"kubernetes.io/projected/966906a4-e943-4f4c-bb84-5f974dff4e85-kube-api-access-qc99w\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.265645 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.265549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/966906a4-e943-4f4c-bb84-5f974dff4e85-signing-cabundle\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.265645 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.265632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/966906a4-e943-4f4c-bb84-5f974dff4e85-signing-key\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.266225 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.266200    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/966906a4-e943-4f4c-bb84-5f974dff4e85-signing-cabundle\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.268374 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.268350    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/966906a4-e943-4f4c-bb84-5f974dff4e85-signing-key\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.275049 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.275024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x6bk7"]
Jul 07 11:40:51.278437 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:51.278385    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b718c89_ec23_49c6_a70a_ebe38d5b482e.slice/crio-21435a0b025f17e9f330f1c8b72e47af2a6e9618be0bf9f486713679b27f9d3d WatchSource:0}: Error finding container 21435a0b025f17e9f330f1c8b72e47af2a6e9618be0bf9f486713679b27f9d3d: Status 404 returned error can't find the container with id 21435a0b025f17e9f330f1c8b72e47af2a6e9618be0bf9f486713679b27f9d3d
Jul 07 11:40:51.284145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.284122    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc99w\" (UniqueName: \"kubernetes.io/projected/966906a4-e943-4f4c-bb84-5f974dff4e85-kube-api-access-qc99w\") pod \"service-ca-758f599c5-4tv9c\" (UID: \"966906a4-e943-4f4c-bb84-5f974dff4e85\") " pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.341103 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.340936    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-4tv9c"
Jul 07 11:40:51.466945 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.466910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:51.467127 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.467010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:51.467127 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:51.467080    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:51.467251 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:51.467161    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls podName:228dd7ca-ee07-481d-bd00-e8648e400bd2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:53.467140344 +0000 UTC m=+38.168073182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qx5d" (UID: "228dd7ca-ee07-481d-bd00-e8648e400bd2") : secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:51.467251 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:51.467164    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 11:40:51.467251 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:51.467221    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls podName:1b7b5a55-0357-4094-b50c-528667fbf4b1 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:53.467201905 +0000 UTC m=+38.168134754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls") pod "dns-default-l5mpr" (UID: "1b7b5a55-0357-4094-b50c-528667fbf4b1") : secret "dns-default-metrics-tls" not found
Jul 07 11:40:51.492989 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.492957    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-4tv9c"]
Jul 07 11:40:51.496416 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:40:51.496391    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966906a4_e943_4f4c_bb84_5f974dff4e85.slice/crio-0c1aa138880fdf729dfd3e3b31f1c8976ffb3ed379b1237a58500b5d7878ed17 WatchSource:0}: Error finding container 0c1aa138880fdf729dfd3e3b31f1c8976ffb3ed379b1237a58500b5d7878ed17: Status 404 returned error can't find the container with id 0c1aa138880fdf729dfd3e3b31f1c8976ffb3ed379b1237a58500b5d7878ed17
Jul 07 11:40:51.549846 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:51.549743    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vcqtg_aa3fdbfa-5535-4d08-a8f0-8052596a9a02/dns-node-resolver/0.log"
Jul 07 11:40:52.093966 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:52.093937    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5h8h7_f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4/node-ca/0.log"
Jul 07 11:40:52.158542 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:52.158507    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc93da87-34c7-4bd9-a16b-dc777abfe795" containerID="ee2dd1811192a38ec7189b7a1b782115b402806465da5d4bec0490d2655a5003" exitCode=0
Jul 07 11:40:52.158719 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:52.158592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerDied","Data":"ee2dd1811192a38ec7189b7a1b782115b402806465da5d4bec0490d2655a5003"}
Jul 07 11:40:52.160034 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:52.160004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-4tv9c" event={"ID":"966906a4-e943-4f4c-bb84-5f974dff4e85","Type":"ContainerStarted","Data":"0c1aa138880fdf729dfd3e3b31f1c8976ffb3ed379b1237a58500b5d7878ed17"}
Jul 07 11:40:52.161231 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:52.161208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6bk7" event={"ID":"8b718c89-ec23-49c6-a70a-ebe38d5b482e","Type":"ContainerStarted","Data":"21435a0b025f17e9f330f1c8b72e47af2a6e9618be0bf9f486713679b27f9d3d"}
Jul 07 11:40:53.167634 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:53.167597    2565 generic.go:356] "Generic (PLEG): container finished" podID="cc93da87-34c7-4bd9-a16b-dc777abfe795" containerID="4f1d44c8dd4eb1c0c319c781356311c3b6e5ebbb447579ba24f9d864a9a8f4f3" exitCode=0
Jul 07 11:40:53.168136 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:53.167642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerDied","Data":"4f1d44c8dd4eb1c0c319c781356311c3b6e5ebbb447579ba24f9d864a9a8f4f3"}
Jul 07 11:40:53.485893 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:53.485800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:53.486040 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:53.485912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:53.486040 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:53.485986    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:53.486040 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:53.486028    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 11:40:53.486146 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:53.486056    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls podName:228dd7ca-ee07-481d-bd00-e8648e400bd2 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:57.486039896 +0000 UTC m=+42.186972733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qx5d" (UID: "228dd7ca-ee07-481d-bd00-e8648e400bd2") : secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:53.486146 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:53.486084    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls podName:1b7b5a55-0357-4094-b50c-528667fbf4b1 nodeName:}" failed. No retries permitted until 2026-07-07 11:40:57.486063336 +0000 UTC m=+42.186996172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls") pod "dns-default-l5mpr" (UID: "1b7b5a55-0357-4094-b50c-528667fbf4b1") : secret "dns-default-metrics-tls" not found
Jul 07 11:40:55.176278 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:55.176238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" event={"ID":"cc93da87-34c7-4bd9-a16b-dc777abfe795","Type":"ContainerStarted","Data":"50251d0604c58982cafe8bee6895bf0222a18f28fd57d1ae32a4f3d9f0e7f164"}
Jul 07 11:40:55.177773 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:55.177744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-4tv9c" event={"ID":"966906a4-e943-4f4c-bb84-5f974dff4e85","Type":"ContainerStarted","Data":"aeb8090e5353a0eaa4f2cd9645448bbb24b0e897865e3c96a2c600ebbed5a4c8"}
Jul 07 11:40:55.179175 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:55.179152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x6bk7" event={"ID":"8b718c89-ec23-49c6-a70a-ebe38d5b482e","Type":"ContainerStarted","Data":"b6b9db2acda2e8182ab35c44e8d13c5d8982d71bfcf2c3f7907f1373b8ecbfdf"}
Jul 07 11:40:55.203821 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:55.203764    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bv5rp" podStartSLOduration=10.714382647 podStartE2EDuration="40.203742912s" totalImagesPullingTime="29.489360265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:55.202145932 +0000 UTC m=+39.903078791" watchObservedRunningTime="2026-07-07 11:40:55.203742912 +0000 UTC m=+39.904675782"
Jul 07 11:40:55.219013 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:55.218962    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x6bk7" podStartSLOduration=3.256494663 podStartE2EDuration="6.218944348s" totalImagesPullingTime="2.962449685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:55.218686591 +0000 UTC m=+39.919619450" watchObservedRunningTime="2026-07-07 11:40:55.218944348 +0000 UTC m=+39.919877207"
Jul 07 11:40:55.237574 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:55.237513    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-4tv9c" podStartSLOduration=1.487969793 podStartE2EDuration="4.237473638s" totalImagesPullingTime="2.749503845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:40:55.236858423 +0000 UTC m=+39.937791285" watchObservedRunningTime="2026-07-07 11:40:55.237473638 +0000 UTC m=+39.938406533"
Jul 07 11:40:57.519831 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:57.519783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:40:57.520283 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:57.519940    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:57.520283 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:40:57.519962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:40:57.520283 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:57.520000    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls podName:228dd7ca-ee07-481d-bd00-e8648e400bd2 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:05.519985113 +0000 UTC m=+50.220917949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4qx5d" (UID: "228dd7ca-ee07-481d-bd00-e8648e400bd2") : secret "insights-runtime-extractor-tls" not found
Jul 07 11:40:57.520283 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:57.520064    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 11:40:57.520283 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:40:57.520120    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls podName:1b7b5a55-0357-4094-b50c-528667fbf4b1 nodeName:}" failed. No retries permitted until 2026-07-07 11:41:05.520103651 +0000 UTC m=+50.221036493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls") pod "dns-default-l5mpr" (UID: "1b7b5a55-0357-4094-b50c-528667fbf4b1") : secret "dns-default-metrics-tls" not found
Jul 07 11:41:05.585191 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.585148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:41:05.585683 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.585222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:41:05.588469 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.588428    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b7b5a55-0357-4094-b50c-528667fbf4b1-metrics-tls\") pod \"dns-default-l5mpr\" (UID: \"1b7b5a55-0357-4094-b50c-528667fbf4b1\") " pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:41:05.588633 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.588504    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/228dd7ca-ee07-481d-bd00-e8648e400bd2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4qx5d\" (UID: \"228dd7ca-ee07-481d-bd00-e8648e400bd2\") " pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:41:05.657029 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.656987    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:41:05.688084 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.688058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4qx5d"
Jul 07 11:41:05.811027 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.810834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l5mpr"]
Jul 07 11:41:05.813412 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:05.813379    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7b5a55_0357_4094_b50c_528667fbf4b1.slice/crio-4d6143a59d90406c55be561a1cf725d0f793686f895cd7ba3f117534c05a158e WatchSource:0}: Error finding container 4d6143a59d90406c55be561a1cf725d0f793686f895cd7ba3f117534c05a158e: Status 404 returned error can't find the container with id 4d6143a59d90406c55be561a1cf725d0f793686f895cd7ba3f117534c05a158e
Jul 07 11:41:05.863112 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:05.863079    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4qx5d"]
Jul 07 11:41:05.866378 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:05.866349    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228dd7ca_ee07_481d_bd00_e8648e400bd2.slice/crio-62457897957a062222e006bfaeec9c6496f1d5f2fef0be4f38cc326039bacd1b WatchSource:0}: Error finding container 62457897957a062222e006bfaeec9c6496f1d5f2fef0be4f38cc326039bacd1b: Status 404 returned error can't find the container with id 62457897957a062222e006bfaeec9c6496f1d5f2fef0be4f38cc326039bacd1b
Jul 07 11:41:06.211057 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:06.210969    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qx5d" event={"ID":"228dd7ca-ee07-481d-bd00-e8648e400bd2","Type":"ContainerStarted","Data":"62457897957a062222e006bfaeec9c6496f1d5f2fef0be4f38cc326039bacd1b"}
Jul 07 11:41:06.212377 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:06.212347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5mpr" event={"ID":"1b7b5a55-0357-4094-b50c-528667fbf4b1","Type":"ContainerStarted","Data":"4d6143a59d90406c55be561a1cf725d0f793686f895cd7ba3f117534c05a158e"}
Jul 07 11:41:11.229253 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:11.229162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qx5d" event={"ID":"228dd7ca-ee07-481d-bd00-e8648e400bd2","Type":"ContainerStarted","Data":"b70aeec3d95582d8f8181a77494514c3567b11f314aa44c5453621387483e1a1"}
Jul 07 11:41:11.229253 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:11.229224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qx5d" event={"ID":"228dd7ca-ee07-481d-bd00-e8648e400bd2","Type":"ContainerStarted","Data":"57c5ac0eb2d343894ef1a4066963fe48a78aa4124db313b9f377e934a093a175"}
Jul 07 11:41:11.230868 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:11.230846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5mpr" event={"ID":"1b7b5a55-0357-4094-b50c-528667fbf4b1","Type":"ContainerStarted","Data":"fac4e792d968df03059d4039393b07f4503f0f277c84ffb3563189852058e93f"}
Jul 07 11:41:11.230963 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:11.230894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5mpr" event={"ID":"1b7b5a55-0357-4094-b50c-528667fbf4b1","Type":"ContainerStarted","Data":"37d803052637525fdb133cbba0c7a8e88588f6c7b73109233f19e419b9228c31"}
Jul 07 11:41:11.230995 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:11.230973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:41:11.260761 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:11.260708    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l5mpr" podStartSLOduration=17.795162169 podStartE2EDuration="22.260691052s" totalImagesPullingTime="4.465528883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:11.25875402 +0000 UTC m=+55.959686880" watchObservedRunningTime="2026-07-07 11:41:11.260691052 +0000 UTC m=+55.961623911"
Jul 07 11:41:12.796663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.796629    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"]
Jul 07 11:41:12.803021 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.802992    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"]
Jul 07 11:41:12.803174 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.803138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.804912 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.804884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.807975 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.807953    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 11:41:12.808928 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.808900    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 11:41:12.809163 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.809144    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 11:41:12.810092 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.810074    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 11:41:12.810206 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.810188    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 11:41:12.810299 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.810276    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 11:41:12.812173 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.812149    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 11:41:12.813101 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.813079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 11:41:12.822473 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.822437    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"]
Jul 07 11:41:12.837116 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.837090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"]
Jul 07 11:41:12.841438 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-ca\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.841626 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488f7054-71cc-4b74-afdb-0e183a96ecc7-tmp\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.841626 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.841626 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841590    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5sqz\" (UniqueName: \"kubernetes.io/projected/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-kube-api-access-c5sqz\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.841626 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dkf6\" (UniqueName: \"kubernetes.io/projected/488f7054-71cc-4b74-afdb-0e183a96ecc7-kube-api-access-4dkf6\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.841841 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-hub\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.841841 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.841948 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/488f7054-71cc-4b74-afdb-0e183a96ecc7-klusterlet-config\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.842006 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.841945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.942440 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942410    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-hub\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.942640 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.942640 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/488f7054-71cc-4b74-afdb-0e183a96ecc7-klusterlet-config\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.942640 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.942640 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-ca\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.942853 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488f7054-71cc-4b74-afdb-0e183a96ecc7-tmp\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.942853 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.942853 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5sqz\" (UniqueName: \"kubernetes.io/projected/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-kube-api-access-c5sqz\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.942853 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.942731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkf6\" (UniqueName: \"kubernetes.io/projected/488f7054-71cc-4b74-afdb-0e183a96ecc7-kube-api-access-4dkf6\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.943507 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.943214    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488f7054-71cc-4b74-afdb-0e183a96ecc7-tmp\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.943507 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.943394    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.945933 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.945885    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-hub\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.946073 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.945931    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.946073 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.945978    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/488f7054-71cc-4b74-afdb-0e183a96ecc7-klusterlet-config\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:12.946194 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.946148    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-ca\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.946936 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.946908    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.962121 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.962092    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5sqz\" (UniqueName: \"kubernetes.io/projected/4b90f7bd-a563-4b1b-b957-9ce93ae2addd-kube-api-access-c5sqz\") pod \"cluster-proxy-proxy-agent-696686c655-9wxp5\" (UID: \"4b90f7bd-a563-4b1b-b957-9ce93ae2addd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:12.966251 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:12.966187    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkf6\" (UniqueName: \"kubernetes.io/projected/488f7054-71cc-4b74-afdb-0e183a96ecc7-kube-api-access-4dkf6\") pod \"klusterlet-addon-workmgr-84d4c97b49-9f2f9\" (UID: \"488f7054-71cc-4b74-afdb-0e183a96ecc7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:13.124169 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:13.124134    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"
Jul 07 11:41:13.131703 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:13.131672    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:13.554434 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:13.554414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5"]
Jul 07 11:41:13.556747 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:13.556723    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b90f7bd_a563_4b1b_b957_9ce93ae2addd.slice/crio-e3cc4615946e8dcc1872f5d28cfbbed59d96218310e8aec60b1f23585a5426f6 WatchSource:0}: Error finding container e3cc4615946e8dcc1872f5d28cfbbed59d96218310e8aec60b1f23585a5426f6: Status 404 returned error can't find the container with id e3cc4615946e8dcc1872f5d28cfbbed59d96218310e8aec60b1f23585a5426f6
Jul 07 11:41:13.567761 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:13.567689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"]
Jul 07 11:41:13.571811 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:13.571788    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488f7054_71cc_4b74_afdb_0e183a96ecc7.slice/crio-0e37f12ca7e10ef5ec86b469cf02792747d6bd70623bf8d36c07cf63ab14dc36 WatchSource:0}: Error finding container 0e37f12ca7e10ef5ec86b469cf02792747d6bd70623bf8d36c07cf63ab14dc36: Status 404 returned error can't find the container with id 0e37f12ca7e10ef5ec86b469cf02792747d6bd70623bf8d36c07cf63ab14dc36
Jul 07 11:41:14.247490 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:14.247424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4qx5d" event={"ID":"228dd7ca-ee07-481d-bd00-e8648e400bd2","Type":"ContainerStarted","Data":"9a183c573411084a1b602697d445c02e913a160dec4efcb50f0b7d1e9dcac41d"}
Jul 07 11:41:14.249188 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:14.249117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9" event={"ID":"488f7054-71cc-4b74-afdb-0e183a96ecc7","Type":"ContainerStarted","Data":"0e37f12ca7e10ef5ec86b469cf02792747d6bd70623bf8d36c07cf63ab14dc36"}
Jul 07 11:41:14.251529 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:14.251501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5" event={"ID":"4b90f7bd-a563-4b1b-b957-9ce93ae2addd","Type":"ContainerStarted","Data":"e3cc4615946e8dcc1872f5d28cfbbed59d96218310e8aec60b1f23585a5426f6"}
Jul 07 11:41:14.270799 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:14.270440    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4qx5d" podStartSLOduration=17.800847967 podStartE2EDuration="25.270419031s" totalImagesPullingTime="7.469571064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:14.270006351 +0000 UTC m=+58.970939211" watchObservedRunningTime="2026-07-07 11:41:14.270419031 +0000 UTC m=+58.971351890"
Jul 07 11:41:15.159475 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:15.158154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l5kdk"
Jul 07 11:41:19.279777 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.279743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9" event={"ID":"488f7054-71cc-4b74-afdb-0e183a96ecc7","Type":"ContainerStarted","Data":"eecc6451e3de060d8dc29b2150ba77b294c13bf906a99889b53466dc5fb4c387"}
Jul 07 11:41:19.280269 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.279837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:19.281838 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.281815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9"
Jul 07 11:41:19.281968 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.281886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5" event={"ID":"4b90f7bd-a563-4b1b-b957-9ce93ae2addd","Type":"ContainerStarted","Data":"3fd1b802896c182619194ab9295e3f4976570ad7680ce605324694707e198f65"}
Jul 07 11:41:19.281968 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.281926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5" event={"ID":"4b90f7bd-a563-4b1b-b957-9ce93ae2addd","Type":"ContainerStarted","Data":"eb086ed98d255474a44617fd05628b09c843e7dd5938d190c91def553f73bc59"}
Jul 07 11:41:19.281968 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.281950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5" event={"ID":"4b90f7bd-a563-4b1b-b957-9ce93ae2addd","Type":"ContainerStarted","Data":"041dd500499669fb155125bc49064c8f3db79c665316e141b013ae1f683b262d"}
Jul 07 11:41:19.298316 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.298265    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84d4c97b49-9f2f9" podStartSLOduration=2.322784853 podStartE2EDuration="7.298250294s" totalImagesPullingTime="4.975465441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:19.297921474 +0000 UTC m=+63.998854345" watchObservedRunningTime="2026-07-07 11:41:19.298250294 +0000 UTC m=+63.999183143"
Jul 07 11:41:19.319224 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:19.319169    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-696686c655-9wxp5" podStartSLOduration=2.330709996 podStartE2EDuration="7.31914873s" totalImagesPullingTime="4.988438734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:19.318522832 +0000 UTC m=+64.019455688" watchObservedRunningTime="2026-07-07 11:41:19.31914873 +0000 UTC m=+64.020081590"
Jul 07 11:41:20.510386 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.510348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:41:20.510795 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.510432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:41:20.513258 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.513241    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 11:41:20.514291 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.514276    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 11:41:20.523703 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.523672    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2-metrics-certs\") pod \"network-metrics-daemon-lrftd\" (UID: \"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2\") " pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:41:20.523816 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.523767    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c996e346-c3a2-4e05-97c9-f3225a44bf73-original-pull-secret\") pod \"global-pull-secret-syncer-85pvm\" (UID: \"c996e346-c3a2-4e05-97c9-f3225a44bf73\") " pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:41:20.611260 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.611220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:41:20.615146 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.615126    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 11:41:20.624446 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.624421    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 11:41:20.634499 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.634446    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l2vw\" (UniqueName: \"kubernetes.io/projected/074e34fb-c8fc-4531-a2f2-4978bdff12d0-kube-api-access-6l2vw\") pod \"network-check-target-82drw\" (UID: \"074e34fb-c8fc-4531-a2f2-4978bdff12d0\") " pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:41:20.794698 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.794614    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vqglz\""
Jul 07 11:41:20.801807 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.801775    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cprtm\""
Jul 07 11:41:20.801953 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.801847    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:41:20.804245 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.804230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-85pvm"
Jul 07 11:41:20.809895 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.809874    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lrftd"
Jul 07 11:41:20.959565 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.959460    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-85pvm"]
Jul 07 11:41:20.963682 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:20.963650    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc996e346_c3a2_4e05_97c9_f3225a44bf73.slice/crio-1a08c260670eb98071ed00301a3069ecd4d8db0946901a25c3e84801b51cd509 WatchSource:0}: Error finding container 1a08c260670eb98071ed00301a3069ecd4d8db0946901a25c3e84801b51cd509: Status 404 returned error can't find the container with id 1a08c260670eb98071ed00301a3069ecd4d8db0946901a25c3e84801b51cd509
Jul 07 11:41:20.965287 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.965259    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-82drw"]
Jul 07 11:41:20.968054 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:20.968028    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074e34fb_c8fc_4531_a2f2_4978bdff12d0.slice/crio-491874481bc9a082193707414648ec0f2dae26b949bafc032cf31e4442753c80 WatchSource:0}: Error finding container 491874481bc9a082193707414648ec0f2dae26b949bafc032cf31e4442753c80: Status 404 returned error can't find the container with id 491874481bc9a082193707414648ec0f2dae26b949bafc032cf31e4442753c80
Jul 07 11:41:20.982141 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:20.982111    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lrftd"]
Jul 07 11:41:20.985603 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:20.985577    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c21f59a_0e3c_4fcb_a2f8_cfe9828c14c2.slice/crio-e85b864427fec23adeca05184e44e2dc13f0fe47c5a75c7372d95b9d8816d134 WatchSource:0}: Error finding container e85b864427fec23adeca05184e44e2dc13f0fe47c5a75c7372d95b9d8816d134: Status 404 returned error can't find the container with id e85b864427fec23adeca05184e44e2dc13f0fe47c5a75c7372d95b9d8816d134
Jul 07 11:41:21.238780 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:21.238747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l5mpr"
Jul 07 11:41:21.289715 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:21.289682    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82drw" event={"ID":"074e34fb-c8fc-4531-a2f2-4978bdff12d0","Type":"ContainerStarted","Data":"491874481bc9a082193707414648ec0f2dae26b949bafc032cf31e4442753c80"}
Jul 07 11:41:21.290974 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:21.290945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lrftd" event={"ID":"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2","Type":"ContainerStarted","Data":"e85b864427fec23adeca05184e44e2dc13f0fe47c5a75c7372d95b9d8816d134"}
Jul 07 11:41:21.292246 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:21.292221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-85pvm" event={"ID":"c996e346-c3a2-4e05-97c9-f3225a44bf73","Type":"ContainerStarted","Data":"1a08c260670eb98071ed00301a3069ecd4d8db0946901a25c3e84801b51cd509"}
Jul 07 11:41:23.306923 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:23.306882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lrftd" event={"ID":"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2","Type":"ContainerStarted","Data":"8be655f59cf7ffaabf1eceb29e2bafc48a85bcad657bda86e0bfd0cfabb42110"}
Jul 07 11:41:23.307472 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:23.307117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lrftd" event={"ID":"6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2","Type":"ContainerStarted","Data":"0abdec311252908c58edebb420fe3b54c56e2fa89299b70caf68a91a68873ba6"}
Jul 07 11:41:23.336526 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:23.336309    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lrftd" podStartSLOduration=65.897882369 podStartE2EDuration="1m7.336286514s" totalImagesPullingTime="1.438404145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:23.33404062 +0000 UTC m=+68.034973502" watchObservedRunningTime="2026-07-07 11:41:23.336286514 +0000 UTC m=+68.037219374"
Jul 07 11:41:26.255272 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.255177    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5nb98"]
Jul 07 11:41:26.257377 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.257355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.260993 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.260971    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 11:41:26.268225 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.268201    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 11:41:26.268665 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.268638    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 11:41:26.268779 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.268693    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 11:41:26.268836 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.268790    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cfrqc\""
Jul 07 11:41:26.268836 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.268790    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 11:41:26.269045 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.269032    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 11:41:26.320696 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.320656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-82drw" event={"ID":"074e34fb-c8fc-4531-a2f2-4978bdff12d0","Type":"ContainerStarted","Data":"a03d576a06479330193c9031a8239ce48c1448ba8f058f3879422a93242ed1e0"}
Jul 07 11:41:26.320882 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.320834    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:41:26.322741 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.322717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-85pvm" event={"ID":"c996e346-c3a2-4e05-97c9-f3225a44bf73","Type":"ContainerStarted","Data":"11e0e002468d6768416cf0b9e2a820687d358ceb2c86a59ad890446c791c0ec2"}
Jul 07 11:41:26.353893 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.353860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-sys\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354069 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.353905    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-tls\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354069 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.353934    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfg7\" (UniqueName: \"kubernetes.io/projected/74962eb8-f3f5-4ed0-9351-9ff787789d48-kube-api-access-ksfg7\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354069 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.353993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-accelerators-collector-config\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354069 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.354051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-root\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354192 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.354094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-textfile\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354192 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.354134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354192 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.354182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74962eb8-f3f5-4ed0-9351-9ff787789d48-metrics-client-ca\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.354282 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.354217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-wtmp\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.391739 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.391674    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-82drw" podStartSLOduration=66.465698637 podStartE2EDuration="1m11.391652863s" totalImagesPullingTime="4.925954226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:26.391495748 +0000 UTC m=+71.092428600" watchObservedRunningTime="2026-07-07 11:41:26.391652863 +0000 UTC m=+71.092585722"
Jul 07 11:41:26.415067 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.414996    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-85pvm" podStartSLOduration=65.473580516 podStartE2EDuration="1m10.414976741s" totalImagesPullingTime="4.941396225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 11:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:26.414182468 +0000 UTC m=+71.115115321" watchObservedRunningTime="2026-07-07 11:41:26.414976741 +0000 UTC m=+71.115909600"
Jul 07 11:41:26.454845 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-sys\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.454845 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-tls\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfg7\" (UniqueName: \"kubernetes.io/projected/74962eb8-f3f5-4ed0-9351-9ff787789d48-kube-api-access-ksfg7\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-accelerators-collector-config\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-root\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454929    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-sys\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-textfile\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.454986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.455019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74962eb8-f3f5-4ed0-9351-9ff787789d48-metrics-client-ca\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455081 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.455057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-wtmp\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455474 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.455116    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-root\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455474 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.455200    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-wtmp\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455474 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.455267    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-textfile\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455788 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.455770    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74962eb8-f3f5-4ed0-9351-9ff787789d48-metrics-client-ca\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.455882 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.455852    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-accelerators-collector-config\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.457380 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.457357    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.457532 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.457514    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/74962eb8-f3f5-4ed0-9351-9ff787789d48-node-exporter-tls\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.467440 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.467412    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfg7\" (UniqueName: \"kubernetes.io/projected/74962eb8-f3f5-4ed0-9351-9ff787789d48-kube-api-access-ksfg7\") pod \"node-exporter-5nb98\" (UID: \"74962eb8-f3f5-4ed0-9351-9ff787789d48\") " pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.566760 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:26.566673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5nb98"
Jul 07 11:41:26.576404 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:41:26.576376    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74962eb8_f3f5_4ed0_9351_9ff787789d48.slice/crio-abc9046472673807d274beaae30c001707a0f07200bf0292214abbc4d361eeba WatchSource:0}: Error finding container abc9046472673807d274beaae30c001707a0f07200bf0292214abbc4d361eeba: Status 404 returned error can't find the container with id abc9046472673807d274beaae30c001707a0f07200bf0292214abbc4d361eeba
Jul 07 11:41:27.328568 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:27.328530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nb98" event={"ID":"74962eb8-f3f5-4ed0-9351-9ff787789d48","Type":"ContainerStarted","Data":"abc9046472673807d274beaae30c001707a0f07200bf0292214abbc4d361eeba"}
Jul 07 11:41:28.332980 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:28.332939    2565 generic.go:356] "Generic (PLEG): container finished" podID="74962eb8-f3f5-4ed0-9351-9ff787789d48" containerID="10dba169a63463cd143b4930d12615581a3dcbeeb677790e5043ad3ae125c5cd" exitCode=0
Jul 07 11:41:28.333407 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:28.333040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nb98" event={"ID":"74962eb8-f3f5-4ed0-9351-9ff787789d48","Type":"ContainerDied","Data":"10dba169a63463cd143b4930d12615581a3dcbeeb677790e5043ad3ae125c5cd"}
Jul 07 11:41:29.339137 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:29.339105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nb98" event={"ID":"74962eb8-f3f5-4ed0-9351-9ff787789d48","Type":"ContainerStarted","Data":"0f8a0d69411226b5985ce728f8976f096867fd9bc07fe5cb71983259c528ca59"}
Jul 07 11:41:29.339734 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:29.339154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5nb98" event={"ID":"74962eb8-f3f5-4ed0-9351-9ff787789d48","Type":"ContainerStarted","Data":"9e2fab709816704318afb5eac1de0dba16733e47b9afa46bf44cedb67fc26b70"}
Jul 07 11:41:29.364077 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:29.363972    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5nb98" podStartSLOduration=2.472378366 podStartE2EDuration="3.363949272s" totalImagesPullingTime="820.683017ms" totalInitContainerRuntime="70.887889ms" isStatefulPod=true podCreationTimestamp="2026-07-07 11:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:41:29.363465897 +0000 UTC m=+74.064398757" watchObservedRunningTime="2026-07-07 11:41:29.363949272 +0000 UTC m=+74.064882132"
Jul 07 11:41:57.331891 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:41:57.331863    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-82drw"
Jul 07 11:42:04.491183 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:42:04.491145    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6bk7_8b718c89-ec23-49c6-a70a-ebe38d5b482e/serve-healthcheck-canary/0.log"
Jul 07 11:45:15.699876 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:15.699849    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:45:15.704317 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:15.704291    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:45:15.707070 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:15.707048    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 11:45:54.222329 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.222296    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-78nm2"]
Jul 07 11:45:54.224319 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.224302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.229649 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.229627    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 07 11:45:54.229963 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.229944    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 07 11:45:54.230103 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.230084    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 07 11:45:54.230728 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.230710    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rdcj4\""
Jul 07 11:45:54.230814 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.230745    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 07 11:45:54.230814 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.230803    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 07 11:45:54.242427 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.242396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-78nm2"]
Jul 07 11:45:54.296552 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.296513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/897af01f-9691-4fd5-bd3a-b5d4cce20327-cabundle0\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.296552 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.296569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvxg6\" (UniqueName: \"kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-kube-api-access-xvxg6\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.296781 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.296663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-certificates\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.397310 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.397269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/897af01f-9691-4fd5-bd3a-b5d4cce20327-cabundle0\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.397310 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.397328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvxg6\" (UniqueName: \"kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-kube-api-access-xvxg6\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.397505 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.397369    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-certificates\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.397505 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:45:54.397405    2565 configmap.go:193] Couldn't get configMap openshift-keda/keda-ocp-cabundle: configmap "keda-ocp-cabundle" not found
Jul 07 11:45:54.397577 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:45:54.397509    2565 secret.go:281] references non-existent secret key: ca.crt
Jul 07 11:45:54.397577 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:45:54.397521    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/897af01f-9691-4fd5-bd3a-b5d4cce20327-cabundle0 podName:897af01f-9691-4fd5-bd3a-b5d4cce20327 nodeName:}" failed. No retries permitted until 2026-07-07 11:45:54.897497168 +0000 UTC m=+339.598430030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cabundle0" (UniqueName: "kubernetes.io/configmap/897af01f-9691-4fd5-bd3a-b5d4cce20327-cabundle0") pod "keda-operator-68bff59c-78nm2" (UID: "897af01f-9691-4fd5-bd3a-b5d4cce20327") : configmap "keda-ocp-cabundle" not found
Jul 07 11:45:54.397577 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:45:54.397525    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 07 11:45:54.397577 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:45:54.397538    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-78nm2: references non-existent secret key: ca.crt
Jul 07 11:45:54.397698 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:45:54.397589    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-certificates podName:897af01f-9691-4fd5-bd3a-b5d4cce20327 nodeName:}" failed. No retries permitted until 2026-07-07 11:45:54.897573236 +0000 UTC m=+339.598506086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-certificates") pod "keda-operator-68bff59c-78nm2" (UID: "897af01f-9691-4fd5-bd3a-b5d4cce20327") : references non-existent secret key: ca.crt
Jul 07 11:45:54.418259 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.418234    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvxg6\" (UniqueName: \"kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-kube-api-access-xvxg6\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.902359 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.902308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-certificates\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.902562 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.902390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/897af01f-9691-4fd5-bd3a-b5d4cce20327-cabundle0\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.902999 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.902978    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/897af01f-9691-4fd5-bd3a-b5d4cce20327-cabundle0\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:54.904816 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:54.904783    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/897af01f-9691-4fd5-bd3a-b5d4cce20327-certificates\") pod \"keda-operator-68bff59c-78nm2\" (UID: \"897af01f-9691-4fd5-bd3a-b5d4cce20327\") " pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:55.134850 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:55.134804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:55.273369 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:55.273337    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-78nm2"]
Jul 07 11:45:55.276978 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:45:55.276943    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897af01f_9691_4fd5_bd3a_b5d4cce20327.slice/crio-7a2578db941a2ecc5424082d4e75733b3d0f5f7ba6225dfb30473811a2646d26 WatchSource:0}: Error finding container 7a2578db941a2ecc5424082d4e75733b3d0f5f7ba6225dfb30473811a2646d26: Status 404 returned error can't find the container with id 7a2578db941a2ecc5424082d4e75733b3d0f5f7ba6225dfb30473811a2646d26
Jul 07 11:45:55.278789 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:55.278771    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:45:56.188066 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:56.188010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-78nm2" event={"ID":"897af01f-9691-4fd5-bd3a-b5d4cce20327","Type":"ContainerStarted","Data":"7a2578db941a2ecc5424082d4e75733b3d0f5f7ba6225dfb30473811a2646d26"}
Jul 07 11:45:59.201065 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:59.201022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-78nm2" event={"ID":"897af01f-9691-4fd5-bd3a-b5d4cce20327","Type":"ContainerStarted","Data":"f9b9d0fa767ce5ccfb1654c534a16572b3d1eac949af85d0f8943c62ff4c00ae"}
Jul 07 11:45:59.201443 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:59.201165    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:45:59.219663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:45:59.219618    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-78nm2" podStartSLOduration=2.070764994 podStartE2EDuration="5.219604592s" totalImagesPullingTime="3.148839598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:45:59.217401684 +0000 UTC m=+343.918334542" watchObservedRunningTime="2026-07-07 11:45:59.219604592 +0000 UTC m=+343.920537451"
Jul 07 11:46:20.207726 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:46:20.207692    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-78nm2"
Jul 07 11:47:01.821654 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.821620    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-9lmzp"]
Jul 07 11:47:01.823749 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.823725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:01.827714 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.827692    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 07 11:47:01.828179 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.828164    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 11:47:01.830098 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.829882    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 11:47:01.830523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.830506    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-4qv45\""
Jul 07 11:47:01.855913 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.855893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205b497e-066a-4674-9521-974ce1d95054-cert\") pod \"kserve-controller-manager-57676b4bd8-9lmzp\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") " pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:01.856009 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.855929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xx7\" (UniqueName: \"kubernetes.io/projected/205b497e-066a-4674-9521-974ce1d95054-kube-api-access-b9xx7\") pod \"kserve-controller-manager-57676b4bd8-9lmzp\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") " pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:01.858512 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.858473    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-9lmzp"]
Jul 07 11:47:01.886391 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.886370    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-hm676"]
Jul 07 11:47:01.888601 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.888586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:01.898975 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.898957    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-89472\""
Jul 07 11:47:01.899308 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.899293    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 11:47:01.915467 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.915438    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-hm676"]
Jul 07 11:47:01.956222 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.956185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xx7\" (UniqueName: \"kubernetes.io/projected/205b497e-066a-4674-9521-974ce1d95054-kube-api-access-b9xx7\") pod \"kserve-controller-manager-57676b4bd8-9lmzp\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") " pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:01.956400 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.956254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205b497e-066a-4674-9521-974ce1d95054-cert\") pod \"kserve-controller-manager-57676b4bd8-9lmzp\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") " pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:01.956400 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.956290    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/974faf44-7afd-4462-8a6d-77d2b0988ad7-data\") pod \"seaweedfs-64568bcd49-hm676\" (UID: \"974faf44-7afd-4462-8a6d-77d2b0988ad7\") " pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:01.956400 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.956322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6wv\" (UniqueName: \"kubernetes.io/projected/974faf44-7afd-4462-8a6d-77d2b0988ad7-kube-api-access-fd6wv\") pod \"seaweedfs-64568bcd49-hm676\" (UID: \"974faf44-7afd-4462-8a6d-77d2b0988ad7\") " pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:01.958622 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.958602    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205b497e-066a-4674-9521-974ce1d95054-cert\") pod \"kserve-controller-manager-57676b4bd8-9lmzp\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") " pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:01.967374 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:01.967352    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xx7\" (UniqueName: \"kubernetes.io/projected/205b497e-066a-4674-9521-974ce1d95054-kube-api-access-b9xx7\") pod \"kserve-controller-manager-57676b4bd8-9lmzp\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") " pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:02.056743 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.056706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/974faf44-7afd-4462-8a6d-77d2b0988ad7-data\") pod \"seaweedfs-64568bcd49-hm676\" (UID: \"974faf44-7afd-4462-8a6d-77d2b0988ad7\") " pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:02.056743 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.056753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6wv\" (UniqueName: \"kubernetes.io/projected/974faf44-7afd-4462-8a6d-77d2b0988ad7-kube-api-access-fd6wv\") pod \"seaweedfs-64568bcd49-hm676\" (UID: \"974faf44-7afd-4462-8a6d-77d2b0988ad7\") " pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:02.057102 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.057081    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/974faf44-7afd-4462-8a6d-77d2b0988ad7-data\") pod \"seaweedfs-64568bcd49-hm676\" (UID: \"974faf44-7afd-4462-8a6d-77d2b0988ad7\") " pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:02.066033 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.066013    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6wv\" (UniqueName: \"kubernetes.io/projected/974faf44-7afd-4462-8a6d-77d2b0988ad7-kube-api-access-fd6wv\") pod \"seaweedfs-64568bcd49-hm676\" (UID: \"974faf44-7afd-4462-8a6d-77d2b0988ad7\") " pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:02.133604 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.133580    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:02.199554 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.199511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:02.280338 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.280308    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-9lmzp"]
Jul 07 11:47:02.283179 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:47:02.283133    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205b497e_066a_4674_9521_974ce1d95054.slice/crio-1ff1a0e987ca70bd83a9967eff2130a9032e8c33d29a7217148dfa0b437b7004 WatchSource:0}: Error finding container 1ff1a0e987ca70bd83a9967eff2130a9032e8c33d29a7217148dfa0b437b7004: Status 404 returned error can't find the container with id 1ff1a0e987ca70bd83a9967eff2130a9032e8c33d29a7217148dfa0b437b7004
Jul 07 11:47:02.330817 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.330785    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-hm676"]
Jul 07 11:47:02.333653 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:47:02.333616    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974faf44_7afd_4462_8a6d_77d2b0988ad7.slice/crio-f191e7e87a210e58411dfc7702c84fb86e5393bfb9101c1d68c2afd53c9a9cd3 WatchSource:0}: Error finding container f191e7e87a210e58411dfc7702c84fb86e5393bfb9101c1d68c2afd53c9a9cd3: Status 404 returned error can't find the container with id f191e7e87a210e58411dfc7702c84fb86e5393bfb9101c1d68c2afd53c9a9cd3
Jul 07 11:47:02.403102 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.403013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp" event={"ID":"205b497e-066a-4674-9521-974ce1d95054","Type":"ContainerStarted","Data":"1ff1a0e987ca70bd83a9967eff2130a9032e8c33d29a7217148dfa0b437b7004"}
Jul 07 11:47:02.404079 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:02.404055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-hm676" event={"ID":"974faf44-7afd-4462-8a6d-77d2b0988ad7","Type":"ContainerStarted","Data":"f191e7e87a210e58411dfc7702c84fb86e5393bfb9101c1d68c2afd53c9a9cd3"}
Jul 07 11:47:06.421687 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:06.421650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp" event={"ID":"205b497e-066a-4674-9521-974ce1d95054","Type":"ContainerStarted","Data":"e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501"}
Jul 07 11:47:06.422120 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:06.421719    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:06.423172 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:06.423149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-hm676" event={"ID":"974faf44-7afd-4462-8a6d-77d2b0988ad7","Type":"ContainerStarted","Data":"a011161130149195bf3db04d046f84e1e4720e6c71a3073047a9ef193c2c5844"}
Jul 07 11:47:06.423297 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:06.423275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:06.442576 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:06.442522    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp" podStartSLOduration=1.905059931 podStartE2EDuration="5.442507543s" totalImagesPullingTime="3.537447612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:47:06.441878836 +0000 UTC m=+411.142811723" watchObservedRunningTime="2026-07-07 11:47:06.442507543 +0000 UTC m=+411.143440403"
Jul 07 11:47:06.460447 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:06.460394    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-hm676" podStartSLOduration=1.886225435 podStartE2EDuration="5.460375617s" totalImagesPullingTime="3.574150182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:47:06.459065842 +0000 UTC m=+411.159998702" watchObservedRunningTime="2026-07-07 11:47:06.460375617 +0000 UTC m=+411.161308476"
Jul 07 11:47:12.429067 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:12.429035    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-hm676"
Jul 07 11:47:37.431866 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:37.431791    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:38.415368 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.415331    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-9lmzp"]
Jul 07 11:47:38.415587 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.415560    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp" podUID="205b497e-066a-4674-9521-974ce1d95054" containerName="manager" containerID="cri-o://e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501" gracePeriod=10
Jul 07 11:47:38.437019 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.436990    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-bm9sm"]
Jul 07 11:47:38.439289 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.439266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.450602 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.450563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-bm9sm"]
Jul 07 11:47:38.526147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.526111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8df\" (UniqueName: \"kubernetes.io/projected/fc229ee2-e63f-45ad-a0a9-af51bd8e3a13-kube-api-access-sb8df\") pod \"kserve-controller-manager-57676b4bd8-bm9sm\" (UID: \"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13\") " pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.526279 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.526190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc229ee2-e63f-45ad-a0a9-af51bd8e3a13-cert\") pod \"kserve-controller-manager-57676b4bd8-bm9sm\" (UID: \"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13\") " pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.626983 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.626948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8df\" (UniqueName: \"kubernetes.io/projected/fc229ee2-e63f-45ad-a0a9-af51bd8e3a13-kube-api-access-sb8df\") pod \"kserve-controller-manager-57676b4bd8-bm9sm\" (UID: \"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13\") " pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.627144 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.627020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc229ee2-e63f-45ad-a0a9-af51bd8e3a13-cert\") pod \"kserve-controller-manager-57676b4bd8-bm9sm\" (UID: \"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13\") " pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.629456 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.629428    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc229ee2-e63f-45ad-a0a9-af51bd8e3a13-cert\") pod \"kserve-controller-manager-57676b4bd8-bm9sm\" (UID: \"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13\") " pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.637126 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.637097    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8df\" (UniqueName: \"kubernetes.io/projected/fc229ee2-e63f-45ad-a0a9-af51bd8e3a13-kube-api-access-sb8df\") pod \"kserve-controller-manager-57676b4bd8-bm9sm\" (UID: \"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13\") " pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.654209 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.654180    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:38.727652 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.727565    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205b497e-066a-4674-9521-974ce1d95054-cert\") pod \"205b497e-066a-4674-9521-974ce1d95054\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") "
Jul 07 11:47:38.727652 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.727630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xx7\" (UniqueName: \"kubernetes.io/projected/205b497e-066a-4674-9521-974ce1d95054-kube-api-access-b9xx7\") pod \"205b497e-066a-4674-9521-974ce1d95054\" (UID: \"205b497e-066a-4674-9521-974ce1d95054\") "
Jul 07 11:47:38.729771 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.729747    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205b497e-066a-4674-9521-974ce1d95054-kube-api-access-b9xx7" (OuterVolumeSpecName: "kube-api-access-b9xx7") pod "205b497e-066a-4674-9521-974ce1d95054" (UID: "205b497e-066a-4674-9521-974ce1d95054"). InnerVolumeSpecName "kube-api-access-b9xx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:47:38.729857 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.729751    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/205b497e-066a-4674-9521-974ce1d95054-cert" (OuterVolumeSpecName: "cert") pod "205b497e-066a-4674-9521-974ce1d95054" (UID: "205b497e-066a-4674-9521-974ce1d95054"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:47:38.793115 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.793076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:38.828275 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.828242    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/205b497e-066a-4674-9521-974ce1d95054-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:47:38.828275 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.828268    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xx7\" (UniqueName: \"kubernetes.io/projected/205b497e-066a-4674-9521-974ce1d95054-kube-api-access-b9xx7\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:47:38.914820 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:38.914792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-bm9sm"]
Jul 07 11:47:38.917084 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:47:38.917058    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc229ee2_e63f_45ad_a0a9_af51bd8e3a13.slice/crio-13ce4deb494b3b2870fa73bab214ebcad515cc0bb392a327dec815b507472971 WatchSource:0}: Error finding container 13ce4deb494b3b2870fa73bab214ebcad515cc0bb392a327dec815b507472971: Status 404 returned error can't find the container with id 13ce4deb494b3b2870fa73bab214ebcad515cc0bb392a327dec815b507472971
Jul 07 11:47:39.532663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.532630    2565 generic.go:356] "Generic (PLEG): container finished" podID="205b497e-066a-4674-9521-974ce1d95054" containerID="e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501" exitCode=0
Jul 07 11:47:39.533101 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.532676    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp"
Jul 07 11:47:39.533101 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.532719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp" event={"ID":"205b497e-066a-4674-9521-974ce1d95054","Type":"ContainerDied","Data":"e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501"}
Jul 07 11:47:39.533101 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.532780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-9lmzp" event={"ID":"205b497e-066a-4674-9521-974ce1d95054","Type":"ContainerDied","Data":"1ff1a0e987ca70bd83a9967eff2130a9032e8c33d29a7217148dfa0b437b7004"}
Jul 07 11:47:39.533101 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.532829    2565 scope.go:117] "RemoveContainer" containerID="e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501"
Jul 07 11:47:39.534305 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.534241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm" event={"ID":"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13","Type":"ContainerStarted","Data":"fc6c69472fc2709d0cdee1e1ec0521861b668bf39297e548c5bbdb2faf2f638e"}
Jul 07 11:47:39.534305 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.534286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm" event={"ID":"fc229ee2-e63f-45ad-a0a9-af51bd8e3a13","Type":"ContainerStarted","Data":"13ce4deb494b3b2870fa73bab214ebcad515cc0bb392a327dec815b507472971"}
Jul 07 11:47:39.534406 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.534380    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:47:39.541831 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.541809    2565 scope.go:117] "RemoveContainer" containerID="e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501"
Jul 07 11:47:39.542065 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:47:39.542045    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501\": container with ID starting with e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501 not found: ID does not exist" containerID="e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501"
Jul 07 11:47:39.542117 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.542072    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501"} err="failed to get container status \"e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501\": rpc error: code = NotFound desc = could not find container \"e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501\": container with ID starting with e59cfbc321dbd6167761957cfdd15fc6cdd44bff4bc889d53f2c22b5d9212501 not found: ID does not exist"
Jul 07 11:47:39.555359 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.555313    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm" podStartSLOduration=1.161288711 podStartE2EDuration="1.555300124s" totalImagesPullingTime="394.011413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:47:39.55441132 +0000 UTC m=+444.255344189" watchObservedRunningTime="2026-07-07 11:47:39.555300124 +0000 UTC m=+444.256232982"
Jul 07 11:47:39.567559 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.567471    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-9lmzp"]
Jul 07 11:47:39.571331 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.571310    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-57676b4bd8-9lmzp"]
Jul 07 11:47:39.884225 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:47:39.884184    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="205b497e-066a-4674-9521-974ce1d95054" path="/var/lib/kubelet/pods/205b497e-066a-4674-9521-974ce1d95054/volumes"
Jul 07 11:48:10.543403 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:10.543370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-57676b4bd8-bm9sm"
Jul 07 11:48:11.503723 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.503695    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-wn4v9"]
Jul 07 11:48:11.503950 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.503938    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205b497e-066a-4674-9521-974ce1d95054" containerName="manager"
Jul 07 11:48:11.503993 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.503952    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="205b497e-066a-4674-9521-974ce1d95054" containerName="manager"
Jul 07 11:48:11.504026 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.503999    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="205b497e-066a-4674-9521-974ce1d95054" containerName="manager"
Jul 07 11:48:11.506865 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.506845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.509592 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.509560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 07 11:48:11.509703 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.509595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-z758x\""
Jul 07 11:48:11.515931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.515910    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-pkmcs"]
Jul 07 11:48:11.519162 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.519143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wn4v9"]
Jul 07 11:48:11.519264 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.519252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:11.522174 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.522154    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-ts6m9\""
Jul 07 11:48:11.522318 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.522160    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 07 11:48:11.529337 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.529319    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-pkmcs"]
Jul 07 11:48:11.662163 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.662132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1c18da9-65da-40e8-9364-b611023521f5-cert\") pod \"odh-model-controller-84b54b7f97-pkmcs\" (UID: \"f1c18da9-65da-40e8-9364-b611023521f5\") " pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:11.662163 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.662172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg66r\" (UniqueName: \"kubernetes.io/projected/f1c18da9-65da-40e8-9364-b611023521f5-kube-api-access-wg66r\") pod \"odh-model-controller-84b54b7f97-pkmcs\" (UID: \"f1c18da9-65da-40e8-9364-b611023521f5\") " pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:11.662626 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.662215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8-tls-certs\") pod \"model-serving-api-fd65d7d6-wn4v9\" (UID: \"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8\") " pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.662626 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.662271    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqpv\" (UniqueName: \"kubernetes.io/projected/19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8-kube-api-access-ftqpv\") pod \"model-serving-api-fd65d7d6-wn4v9\" (UID: \"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8\") " pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.763659 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.763574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1c18da9-65da-40e8-9364-b611023521f5-cert\") pod \"odh-model-controller-84b54b7f97-pkmcs\" (UID: \"f1c18da9-65da-40e8-9364-b611023521f5\") " pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:11.763659 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.763611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg66r\" (UniqueName: \"kubernetes.io/projected/f1c18da9-65da-40e8-9364-b611023521f5-kube-api-access-wg66r\") pod \"odh-model-controller-84b54b7f97-pkmcs\" (UID: \"f1c18da9-65da-40e8-9364-b611023521f5\") " pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:11.763659 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.763643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8-tls-certs\") pod \"model-serving-api-fd65d7d6-wn4v9\" (UID: \"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8\") " pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.763937 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.763677    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqpv\" (UniqueName: \"kubernetes.io/projected/19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8-kube-api-access-ftqpv\") pod \"model-serving-api-fd65d7d6-wn4v9\" (UID: \"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8\") " pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.763937 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:48:11.763744    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 07 11:48:11.763937 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:48:11.763821    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1c18da9-65da-40e8-9364-b611023521f5-cert podName:f1c18da9-65da-40e8-9364-b611023521f5 nodeName:}" failed. No retries permitted until 2026-07-07 11:48:12.263799134 +0000 UTC m=+476.964731970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f1c18da9-65da-40e8-9364-b611023521f5-cert") pod "odh-model-controller-84b54b7f97-pkmcs" (UID: "f1c18da9-65da-40e8-9364-b611023521f5") : secret "odh-model-controller-webhook-cert" not found
Jul 07 11:48:11.766047 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.766021    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8-tls-certs\") pod \"model-serving-api-fd65d7d6-wn4v9\" (UID: \"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8\") " pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.774927 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.774903    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg66r\" (UniqueName: \"kubernetes.io/projected/f1c18da9-65da-40e8-9364-b611023521f5-kube-api-access-wg66r\") pod \"odh-model-controller-84b54b7f97-pkmcs\" (UID: \"f1c18da9-65da-40e8-9364-b611023521f5\") " pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:11.775020 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.774989    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqpv\" (UniqueName: \"kubernetes.io/projected/19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8-kube-api-access-ftqpv\") pod \"model-serving-api-fd65d7d6-wn4v9\" (UID: \"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8\") " pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.817519 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.817471    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:11.940998 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:11.940954    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wn4v9"]
Jul 07 11:48:11.943420 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:48:11.943396    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f5c206_cf44_4e3e_9d9f_fe8fc7c052e8.slice/crio-5e2b9ccf3cb18e266b05e063e93fe9ba3a17f75a457f580308ab854cf44c5cf6 WatchSource:0}: Error finding container 5e2b9ccf3cb18e266b05e063e93fe9ba3a17f75a457f580308ab854cf44c5cf6: Status 404 returned error can't find the container with id 5e2b9ccf3cb18e266b05e063e93fe9ba3a17f75a457f580308ab854cf44c5cf6
Jul 07 11:48:12.268080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:12.268046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1c18da9-65da-40e8-9364-b611023521f5-cert\") pod \"odh-model-controller-84b54b7f97-pkmcs\" (UID: \"f1c18da9-65da-40e8-9364-b611023521f5\") " pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:12.270358 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:12.270332    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1c18da9-65da-40e8-9364-b611023521f5-cert\") pod \"odh-model-controller-84b54b7f97-pkmcs\" (UID: \"f1c18da9-65da-40e8-9364-b611023521f5\") " pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:12.431016 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:12.430976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:12.555597 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:12.555524    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-pkmcs"]
Jul 07 11:48:12.558107 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:48:12.558083    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c18da9_65da_40e8_9364_b611023521f5.slice/crio-69a04f14875ec709a54500c916178fcd8fc53ed6b41804f22e1749132b3a685f WatchSource:0}: Error finding container 69a04f14875ec709a54500c916178fcd8fc53ed6b41804f22e1749132b3a685f: Status 404 returned error can't find the container with id 69a04f14875ec709a54500c916178fcd8fc53ed6b41804f22e1749132b3a685f
Jul 07 11:48:12.646474 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:12.646438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-pkmcs" event={"ID":"f1c18da9-65da-40e8-9364-b611023521f5","Type":"ContainerStarted","Data":"69a04f14875ec709a54500c916178fcd8fc53ed6b41804f22e1749132b3a685f"}
Jul 07 11:48:12.647528 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:12.647503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wn4v9" event={"ID":"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8","Type":"ContainerStarted","Data":"5e2b9ccf3cb18e266b05e063e93fe9ba3a17f75a457f580308ab854cf44c5cf6"}
Jul 07 11:48:14.657797 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:14.657755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wn4v9" event={"ID":"19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8","Type":"ContainerStarted","Data":"266b729088cbf2066430e246d1e97cb62a78488384b5d945eb38efcbeb7cb511"}
Jul 07 11:48:14.658249 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:14.657901    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:14.675718 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:14.675671    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-wn4v9" podStartSLOduration=1.931953154 podStartE2EDuration="3.675656631s" totalImagesPullingTime="1.743703477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:48:14.673408286 +0000 UTC m=+479.374341147" watchObservedRunningTime="2026-07-07 11:48:14.675656631 +0000 UTC m=+479.376589489"
Jul 07 11:48:15.663907 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:15.663870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-pkmcs" event={"ID":"f1c18da9-65da-40e8-9364-b611023521f5","Type":"ContainerStarted","Data":"547963dcfca71fccbdecd6292823ce6abf75bc9730a24116c652ddae29ab9075"}
Jul 07 11:48:15.664418 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:15.664001    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:48:15.681144 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:15.681068    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-pkmcs" podStartSLOduration=1.9263841080000002 podStartE2EDuration="4.681051216s" totalImagesPullingTime="2.754667108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:48:15.680068872 +0000 UTC m=+480.381001730" watchObservedRunningTime="2026-07-07 11:48:15.681051216 +0000 UTC m=+480.381984072"
Jul 07 11:48:25.667873 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:25.667841    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-wn4v9"
Jul 07 11:48:26.669349 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:48:26.669318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-pkmcs"
Jul 07 11:49:57.898386 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.898358    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"]
Jul 07 11:49:57.901702 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.901683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:57.904290 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.904269    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 11:49:57.904511 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.904271    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 11:49:57.904511 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.904303    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lhlhn\""
Jul 07 11:49:57.904663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.904338    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a3c05-kube-rbac-proxy-sar-config\""
Jul 07 11:49:57.904663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.904341    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 11:49:57.904663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.904366    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 11:49:57.904663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.904387    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a3c05-predictor-serving-cert\""
Jul 07 11:49:57.915711 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:57.915689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"]
Jul 07 11:49:58.004862 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.004830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x298g\" (UniqueName: \"kubernetes.io/projected/aea60b20-183b-450f-b3ff-3946a365f495-kube-api-access-x298g\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.004862 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.004869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.005079 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.004887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aea60b20-183b-450f-b3ff-3946a365f495-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.005079 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.004939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aea60b20-183b-450f-b3ff-3946a365f495-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.005079 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.005020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.106405 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.106373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aea60b20-183b-450f-b3ff-3946a365f495-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.106614 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.106416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.106680 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.106636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x298g\" (UniqueName: \"kubernetes.io/projected/aea60b20-183b-450f-b3ff-3946a365f495-kube-api-access-x298g\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.106749 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.106689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.106749 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.106737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aea60b20-183b-450f-b3ff-3946a365f495-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.107179 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.107143    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aea60b20-183b-450f-b3ff-3946a365f495-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.107382 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.107362    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.107464 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.107398    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.108998 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.108977    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aea60b20-183b-450f-b3ff-3946a365f495-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.115556 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.115534    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x298g\" (UniqueName: \"kubernetes.io/projected/aea60b20-183b-450f-b3ff-3946a365f495-kube-api-access-x298g\") pod \"isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.212690 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.212608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:49:58.354126 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:58.354097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"]
Jul 07 11:49:58.357400 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:49:58.357371    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea60b20_183b_450f_b3ff_3946a365f495.slice/crio-79ed4c980ed1956b2896ab6daec1d5a0ae0df1e4af9ba96b89b98704f88be117 WatchSource:0}: Error finding container 79ed4c980ed1956b2896ab6daec1d5a0ae0df1e4af9ba96b89b98704f88be117: Status 404 returned error can't find the container with id 79ed4c980ed1956b2896ab6daec1d5a0ae0df1e4af9ba96b89b98704f88be117
Jul 07 11:49:59.014381 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:49:59.014346    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerStarted","Data":"79ed4c980ed1956b2896ab6daec1d5a0ae0df1e4af9ba96b89b98704f88be117"}
Jul 07 11:50:02.027858 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:02.027778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerStarted","Data":"8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654"}
Jul 07 11:50:06.043501 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:06.043440    2565 generic.go:356] "Generic (PLEG): container finished" podID="aea60b20-183b-450f-b3ff-3946a365f495" containerID="8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654" exitCode=0
Jul 07 11:50:06.043916 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:06.043516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerDied","Data":"8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654"}
Jul 07 11:50:19.296838 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:19.296802    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:50:19.297609 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:19.297585    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:50:20.115461 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:20.115419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerStarted","Data":"11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0"}
Jul 07 11:50:22.124956 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:22.124916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerStarted","Data":"0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc"}
Jul 07 11:50:25.137383 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:25.137344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerStarted","Data":"3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065"}
Jul 07 11:50:25.137802 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:25.137624    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:50:25.137802 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:25.137700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:50:25.138943 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:25.138910    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:25.161560 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:25.161509    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podStartSLOduration=2.4060652559999998 podStartE2EDuration="28.161490975s" totalImagesPullingTime="22.097002947s" totalInitContainerRuntime="3.658422772s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:49:57 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:50:25.159763561 +0000 UTC m=+609.860696421" watchObservedRunningTime="2026-07-07 11:50:25.161490975 +0000 UTC m=+609.862423825"
Jul 07 11:50:26.141252 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:26.141215    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:50:26.141769 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:26.141336    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:26.142409 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:26.142379    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:50:26.145165 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:26.145146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:50:27.144740 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:27.144690    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:27.145196 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:27.145090    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:50:28.148356 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:28.148318    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:28.148782 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:28.148682    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:50:38.148863 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:38.148763    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:38.149336 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:38.149321    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:50:48.148725 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:48.148678    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:48.149145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:48.149081    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:50:58.148500 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:58.148446    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:50:58.149042 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:50:58.148891    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:08.148848 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:08.148800    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:51:08.149250 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:08.149223    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:18.148736 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:18.148684    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:51:18.149248 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:18.149163    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:28.148889 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:28.148842    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:51:28.149323 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:28.149301    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:38.149529 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:38.149466    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:51:38.149953 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:38.149657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:51:53.021754 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.021715    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"]
Jul 07 11:51:53.022307 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.022052    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" containerID="cri-o://11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0" gracePeriod=30
Jul 07 11:51:53.022307 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.022071    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" containerID="cri-o://3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065" gracePeriod=30
Jul 07 11:51:53.022307 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.022098    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy" containerID="cri-o://0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc" gracePeriod=30
Jul 07 11:51:53.037977 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.037933    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"]
Jul 07 11:51:53.041261 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.041242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.043753 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.043730    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-fea16-predictor-serving-cert\""
Jul 07 11:51:53.043849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.043762    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-fea16-kube-rbac-proxy-sar-config\""
Jul 07 11:51:53.055726 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.055703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"]
Jul 07 11:51:53.102084 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.102049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllsn\" (UniqueName: \"kubernetes.io/projected/c15d4442-c580-40c0-b82e-621022196d32-kube-api-access-bllsn\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.102084 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.102088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15d4442-c580-40c0-b82e-621022196d32-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.102322 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.102188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.102322 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.102245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.102322 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.102294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-cabundle-cert\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.203030 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.202978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllsn\" (UniqueName: \"kubernetes.io/projected/c15d4442-c580-40c0-b82e-621022196d32-kube-api-access-bllsn\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.203030 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.203025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15d4442-c580-40c0-b82e-621022196d32-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.203273 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.203073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.203273 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.203115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.203273 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.203157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-cabundle-cert\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.203420 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:51:53.203313    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-serving-cert: secret "isvc-sklearn-graph-raw-fea16-predictor-serving-cert" not found
Jul 07 11:51:53.203420 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:51:53.203400    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls podName:c15d4442-c580-40c0-b82e-621022196d32 nodeName:}" failed. No retries permitted until 2026-07-07 11:51:53.703377474 +0000 UTC m=+698.404310331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls") pod "isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" (UID: "c15d4442-c580-40c0-b82e-621022196d32") : secret "isvc-sklearn-graph-raw-fea16-predictor-serving-cert" not found
Jul 07 11:51:53.203599 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.203563    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15d4442-c580-40c0-b82e-621022196d32-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.203976 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.203954    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-cabundle-cert\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.204020 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.203982    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.214577 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.214541    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllsn\" (UniqueName: \"kubernetes.io/projected/c15d4442-c580-40c0-b82e-621022196d32-kube-api-access-bllsn\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.447852 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.447819    2565 generic.go:356] "Generic (PLEG): container finished" podID="aea60b20-183b-450f-b3ff-3946a365f495" containerID="0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc" exitCode=2
Jul 07 11:51:53.448010 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.447881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerDied","Data":"0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc"}
Jul 07 11:51:53.708551 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.708437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.710839 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.710812    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls\") pod \"isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:53.955723 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:53.955691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:54.082918 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:54.082560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"]
Jul 07 11:51:54.085220 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:51:54.085188    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15d4442_c580_40c0_b82e_621022196d32.slice/crio-20497cd27cf8763e22c97f0deeb2cb5b2104e32af1e0ba9ef47ea0f410a8bad6 WatchSource:0}: Error finding container 20497cd27cf8763e22c97f0deeb2cb5b2104e32af1e0ba9ef47ea0f410a8bad6: Status 404 returned error can't find the container with id 20497cd27cf8763e22c97f0deeb2cb5b2104e32af1e0ba9ef47ea0f410a8bad6
Jul 07 11:51:54.452767 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:54.452726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerStarted","Data":"dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594"}
Jul 07 11:51:54.452971 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:54.452787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerStarted","Data":"20497cd27cf8763e22c97f0deeb2cb5b2104e32af1e0ba9ef47ea0f410a8bad6"}
Jul 07 11:51:56.141765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:56.141711    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 11:51:58.148405 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:58.148356    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:51:58.148849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:58.148723    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:51:58.467985 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:58.467954    2565 generic.go:356] "Generic (PLEG): container finished" podID="c15d4442-c580-40c0-b82e-621022196d32" containerID="dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594" exitCode=0
Jul 07 11:51:58.468196 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:58.468028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerDied","Data":"dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594"}
Jul 07 11:51:58.470493 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:58.470459    2565 generic.go:356] "Generic (PLEG): container finished" podID="aea60b20-183b-450f-b3ff-3946a365f495" containerID="11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0" exitCode=0
Jul 07 11:51:58.470600 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:58.470510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerDied","Data":"11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0"}
Jul 07 11:51:59.476597 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:59.476563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerStarted","Data":"ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e"}
Jul 07 11:51:59.476998 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:59.476621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerStarted","Data":"8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5"}
Jul 07 11:51:59.476998 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:59.476881    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:59.477103 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:59.477011    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:51:59.478362 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:59.478337    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:51:59.499431 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:51:59.499390    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podStartSLOduration=2.687073589 podStartE2EDuration="6.499378224s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.812304635s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:51:59.497931632 +0000 UTC m=+704.198864488" watchObservedRunningTime="2026-07-07 11:51:59.499378224 +0000 UTC m=+704.200311083"
Jul 07 11:52:00.479986 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:00.479949    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:52:01.141917 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:01.141877    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 11:52:05.484599 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:05.484567    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:52:05.485073 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:05.485024    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:52:06.142144 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:06.142100    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 11:52:06.142322 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:06.142235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:52:08.148818 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:08.148776    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:52:08.149512 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:08.149047    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:11.141540 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:11.141497    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 11:52:15.485302 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:15.485267    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:52:16.142463 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:16.142419    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 11:52:18.149256 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:18.149202    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 07 11:52:18.149739 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:18.149364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:52:18.149739 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:18.149554    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:52:18.149739 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:18.149647    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:52:21.142265 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:21.142221    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 07 11:52:23.173583 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.173556    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:52:23.255595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.255560    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x298g\" (UniqueName: \"kubernetes.io/projected/aea60b20-183b-450f-b3ff-3946a365f495-kube-api-access-x298g\") pod \"aea60b20-183b-450f-b3ff-3946a365f495\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") "
Jul 07 11:52:23.255595 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.255600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-kube-rbac-proxy-sar-config\") pod \"aea60b20-183b-450f-b3ff-3946a365f495\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") "
Jul 07 11:52:23.255820 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.255627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-cabundle-cert\") pod \"aea60b20-183b-450f-b3ff-3946a365f495\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") "
Jul 07 11:52:23.255820 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.255658    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aea60b20-183b-450f-b3ff-3946a365f495-kserve-provision-location\") pod \"aea60b20-183b-450f-b3ff-3946a365f495\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") "
Jul 07 11:52:23.255820 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.255690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aea60b20-183b-450f-b3ff-3946a365f495-proxy-tls\") pod \"aea60b20-183b-450f-b3ff-3946a365f495\" (UID: \"aea60b20-183b-450f-b3ff-3946a365f495\") "
Jul 07 11:52:23.256009 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.255977    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea60b20-183b-450f-b3ff-3946a365f495-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aea60b20-183b-450f-b3ff-3946a365f495" (UID: "aea60b20-183b-450f-b3ff-3946a365f495"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:52:23.256119 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.256002    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aea60b20-183b-450f-b3ff-3946a365f495" (UID: "aea60b20-183b-450f-b3ff-3946a365f495"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:23.256161 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.256112    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aea60b20-183b-450f-b3ff-3946a365f495" (UID: "aea60b20-183b-450f-b3ff-3946a365f495"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:52:23.257884 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.257863    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea60b20-183b-450f-b3ff-3946a365f495-kube-api-access-x298g" (OuterVolumeSpecName: "kube-api-access-x298g") pod "aea60b20-183b-450f-b3ff-3946a365f495" (UID: "aea60b20-183b-450f-b3ff-3946a365f495"). InnerVolumeSpecName "kube-api-access-x298g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:52:23.257974 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.257895    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea60b20-183b-450f-b3ff-3946a365f495-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aea60b20-183b-450f-b3ff-3946a365f495" (UID: "aea60b20-183b-450f-b3ff-3946a365f495"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:52:23.356935 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.356904    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x298g\" (UniqueName: \"kubernetes.io/projected/aea60b20-183b-450f-b3ff-3946a365f495-kube-api-access-x298g\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:52:23.356935 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.356934    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:52:23.357121 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.356945    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aea60b20-183b-450f-b3ff-3946a365f495-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:52:23.357121 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.356953    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aea60b20-183b-450f-b3ff-3946a365f495-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:52:23.357121 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.356963    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aea60b20-183b-450f-b3ff-3946a365f495-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:52:23.562983 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.562950    2565 generic.go:356] "Generic (PLEG): container finished" podID="aea60b20-183b-450f-b3ff-3946a365f495" containerID="3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065" exitCode=0
Jul 07 11:52:23.563149 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.563022    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"
Jul 07 11:52:23.563149 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.563019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerDied","Data":"3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065"}
Jul 07 11:52:23.563149 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.563112    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg" event={"ID":"aea60b20-183b-450f-b3ff-3946a365f495","Type":"ContainerDied","Data":"79ed4c980ed1956b2896ab6daec1d5a0ae0df1e4af9ba96b89b98704f88be117"}
Jul 07 11:52:23.563149 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.563142    2565 scope.go:117] "RemoveContainer" containerID="3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065"
Jul 07 11:52:23.571391 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.571373    2565 scope.go:117] "RemoveContainer" containerID="0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc"
Jul 07 11:52:23.578304 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.578290    2565 scope.go:117] "RemoveContainer" containerID="11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0"
Jul 07 11:52:23.585254 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.585238    2565 scope.go:117] "RemoveContainer" containerID="8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654"
Jul 07 11:52:23.585330 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.585270    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"]
Jul 07 11:52:23.591716 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.591691    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a3c05-predictor-6dbbcd969d-xnjtg"]
Jul 07 11:52:23.592244 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.592231    2565 scope.go:117] "RemoveContainer" containerID="3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065"
Jul 07 11:52:23.592498 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:52:23.592466    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065\": container with ID starting with 3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065 not found: ID does not exist" containerID="3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065"
Jul 07 11:52:23.592543 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.592510    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065"} err="failed to get container status \"3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065\": rpc error: code = NotFound desc = could not find container \"3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065\": container with ID starting with 3097df86f7f12e4ba7885e700d6c38f0c127cba8cdd06a9ad2dc43a64479c065 not found: ID does not exist"
Jul 07 11:52:23.592543 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.592533    2565 scope.go:117] "RemoveContainer" containerID="0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc"
Jul 07 11:52:23.592766 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:52:23.592750    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc\": container with ID starting with 0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc not found: ID does not exist" containerID="0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc"
Jul 07 11:52:23.592808 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.592771    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc"} err="failed to get container status \"0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc\": rpc error: code = NotFound desc = could not find container \"0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc\": container with ID starting with 0ef14c26bb0d2861a0ea798604ff0450325370a673a5fd2fdfefbbaa49f4fcdc not found: ID does not exist"
Jul 07 11:52:23.592808 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.592788    2565 scope.go:117] "RemoveContainer" containerID="11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0"
Jul 07 11:52:23.593005 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:52:23.592991    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0\": container with ID starting with 11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0 not found: ID does not exist" containerID="11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0"
Jul 07 11:52:23.593046 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.593009    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0"} err="failed to get container status \"11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0\": rpc error: code = NotFound desc = could not find container \"11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0\": container with ID starting with 11ae1dcf37ae8db5a02d496e90c0477924e827ec65c65976ef17c85edf092bc0 not found: ID does not exist"
Jul 07 11:52:23.593046 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.593020    2565 scope.go:117] "RemoveContainer" containerID="8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654"
Jul 07 11:52:23.593209 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:52:23.593192    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654\": container with ID starting with 8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654 not found: ID does not exist" containerID="8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654"
Jul 07 11:52:23.593246 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.593217    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654"} err="failed to get container status \"8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654\": rpc error: code = NotFound desc = could not find container \"8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654\": container with ID starting with 8284693cd09e4b4f0c1a9e1eafb8660d561b99376227ec9a2ed674c320234654 not found: ID does not exist"
Jul 07 11:52:23.884171 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:23.884141    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea60b20-183b-450f-b3ff-3946a365f495" path="/var/lib/kubelet/pods/aea60b20-183b-450f-b3ff-3946a365f495/volumes"
Jul 07 11:52:25.485500 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:25.485447    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:52:35.485165 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:35.485126    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:52:45.485676 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:45.485632    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:52:55.485782 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:52:55.485740    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:53:05.485261 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:05.485223    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:53:15.485633 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:15.485602    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:53:33.062050 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062022    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"]
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062293    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062302    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062315    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062321    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062327    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062334    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062343    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="storage-initializer"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062348    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="storage-initializer"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062388    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kserve-container"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062395    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="kube-rbac-proxy"
Jul 07 11:53:33.062497 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.062401    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea60b20-183b-450f-b3ff-3946a365f495" containerName="agent"
Jul 07 11:53:33.066348 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.066327    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.068941 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.068923    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-fea16-kube-rbac-proxy-sar-config\""
Jul 07 11:53:33.069111 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.069089    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-fea16-serving-cert\""
Jul 07 11:53:33.074094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.074072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"]
Jul 07 11:53:33.183360 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.183324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls\") pod \"model-chainer-raw-fea16-7bbdc5b5df-wcghl\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") " pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.183588 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.183370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926403a4-3d20-469a-b75f-cecaccfcdf03-openshift-service-ca-bundle\") pod \"model-chainer-raw-fea16-7bbdc5b5df-wcghl\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") " pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.284756 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.284721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls\") pod \"model-chainer-raw-fea16-7bbdc5b5df-wcghl\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") " pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.284756 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.284762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926403a4-3d20-469a-b75f-cecaccfcdf03-openshift-service-ca-bundle\") pod \"model-chainer-raw-fea16-7bbdc5b5df-wcghl\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") " pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.284997 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:53:33.284869    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-fea16-serving-cert: secret "model-chainer-raw-fea16-serving-cert" not found
Jul 07 11:53:33.284997 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:53:33.284943    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls podName:926403a4-3d20-469a-b75f-cecaccfcdf03 nodeName:}" failed. No retries permitted until 2026-07-07 11:53:33.784925555 +0000 UTC m=+798.485858391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls") pod "model-chainer-raw-fea16-7bbdc5b5df-wcghl" (UID: "926403a4-3d20-469a-b75f-cecaccfcdf03") : secret "model-chainer-raw-fea16-serving-cert" not found
Jul 07 11:53:33.285322 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.285304    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926403a4-3d20-469a-b75f-cecaccfcdf03-openshift-service-ca-bundle\") pod \"model-chainer-raw-fea16-7bbdc5b5df-wcghl\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") " pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.789372 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.789328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls\") pod \"model-chainer-raw-fea16-7bbdc5b5df-wcghl\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") " pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.791734 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.791713    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls\") pod \"model-chainer-raw-fea16-7bbdc5b5df-wcghl\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") " pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:33.977546 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:33.977502    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:34.107162 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:34.107133    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"]
Jul 07 11:53:34.111150 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:53:34.111116    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926403a4_3d20_469a_b75f_cecaccfcdf03.slice/crio-3f528acb200c7f64f88166a79613e7bed9519fc3f543af7ea821ff96f53738ba WatchSource:0}: Error finding container 3f528acb200c7f64f88166a79613e7bed9519fc3f543af7ea821ff96f53738ba: Status 404 returned error can't find the container with id 3f528acb200c7f64f88166a79613e7bed9519fc3f543af7ea821ff96f53738ba
Jul 07 11:53:34.113092 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:34.113072    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 11:53:34.820498 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:34.820446    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" event={"ID":"926403a4-3d20-469a-b75f-cecaccfcdf03","Type":"ContainerStarted","Data":"3f528acb200c7f64f88166a79613e7bed9519fc3f543af7ea821ff96f53738ba"}
Jul 07 11:53:36.828569 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:36.828527    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" event={"ID":"926403a4-3d20-469a-b75f-cecaccfcdf03","Type":"ContainerStarted","Data":"3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c"}
Jul 07 11:53:36.828955 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:36.828633    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:36.847305 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:36.847257    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podStartSLOduration=1.7877894300000001 podStartE2EDuration="3.847242558s" totalImagesPullingTime="2.059453128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:53:36.846188718 +0000 UTC m=+801.547121577" watchObservedRunningTime="2026-07-07 11:53:36.847242558 +0000 UTC m=+801.548175414"
Jul 07 11:53:42.838136 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:42.838109    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:53:43.118365 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.118333    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"]
Jul 07 11:53:43.118587 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.118562    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16" containerID="cri-o://3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c" gracePeriod=30
Jul 07 11:53:43.358901 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.358561    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"]
Jul 07 11:53:43.367678 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.366309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.367678 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.367003    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"]
Jul 07 11:53:43.367678 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.367324    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" containerID="cri-o://8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5" gracePeriod=30
Jul 07 11:53:43.367678 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.367381    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kube-rbac-proxy" containerID="cri-o://ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e" gracePeriod=30
Jul 07 11:53:43.369954 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.369929    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-serving-cert\""
Jul 07 11:53:43.370302 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.370162    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-e5dd1-kube-rbac-proxy-sar-config\""
Jul 07 11:53:43.378471 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.378445    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"]
Jul 07 11:53:43.470318 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.470280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rh9\" (UniqueName: \"kubernetes.io/projected/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-api-access-26rh9\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.470611 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.470578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.470781 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.470726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.470877 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.470832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.470935 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.470897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.571660 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.571621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.571849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.571681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.571849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.571715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rh9\" (UniqueName: \"kubernetes.io/projected/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-api-access-26rh9\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.571849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.571756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.571849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.571812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.572186 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.572161    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.572359 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.572335    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.572472 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.572455    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.574352 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.574322    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.584311 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.584288    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rh9\" (UniqueName: \"kubernetes.io/projected/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-api-access-26rh9\") pod \"isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.686065 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.685982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:43.828876 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.828851    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"]
Jul 07 11:53:43.831469 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:53:43.831444    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2935c9c_9c4c_4da4_9cf2_72d78d558ca8.slice/crio-a3364d81cf332b1a4a4b53842b7d833115b375655afdbeda768d2989c70fed04 WatchSource:0}: Error finding container a3364d81cf332b1a4a4b53842b7d833115b375655afdbeda768d2989c70fed04: Status 404 returned error can't find the container with id a3364d81cf332b1a4a4b53842b7d833115b375655afdbeda768d2989c70fed04
Jul 07 11:53:43.854665 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.854635    2565 generic.go:356] "Generic (PLEG): container finished" podID="c15d4442-c580-40c0-b82e-621022196d32" containerID="ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e" exitCode=2
Jul 07 11:53:43.854998 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.854717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerDied","Data":"ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e"}
Jul 07 11:53:43.855892 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:43.855870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerStarted","Data":"a3364d81cf332b1a4a4b53842b7d833115b375655afdbeda768d2989c70fed04"}
Jul 07 11:53:44.861241 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:44.861204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerStarted","Data":"9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4"}
Jul 07 11:53:45.480875 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:45.480825    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 07 11:53:45.485153 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:45.485127    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 07 11:53:47.836904 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:47.836823    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:47.878564 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:47.878532    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerID="9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4" exitCode=0
Jul 07 11:53:47.878682 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:47.878606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerDied","Data":"9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4"}
Jul 07 11:53:48.036660 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.036633    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:53:48.105236 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105208    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15d4442-c580-40c0-b82e-621022196d32-kserve-provision-location\") pod \"c15d4442-c580-40c0-b82e-621022196d32\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") "
Jul 07 11:53:48.105384 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bllsn\" (UniqueName: \"kubernetes.io/projected/c15d4442-c580-40c0-b82e-621022196d32-kube-api-access-bllsn\") pod \"c15d4442-c580-40c0-b82e-621022196d32\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") "
Jul 07 11:53:48.105384 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105291    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-cabundle-cert\") pod \"c15d4442-c580-40c0-b82e-621022196d32\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") "
Jul 07 11:53:48.105384 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-kube-rbac-proxy-sar-config\") pod \"c15d4442-c580-40c0-b82e-621022196d32\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") "
Jul 07 11:53:48.105384 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105350    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls\") pod \"c15d4442-c580-40c0-b82e-621022196d32\" (UID: \"c15d4442-c580-40c0-b82e-621022196d32\") "
Jul 07 11:53:48.105621 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105608    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15d4442-c580-40c0-b82e-621022196d32-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c15d4442-c580-40c0-b82e-621022196d32" (UID: "c15d4442-c580-40c0-b82e-621022196d32"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:53:48.105715 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105688    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c15d4442-c580-40c0-b82e-621022196d32" (UID: "c15d4442-c580-40c0-b82e-621022196d32"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:53:48.105813 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.105686    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c15d4442-c580-40c0-b82e-621022196d32" (UID: "c15d4442-c580-40c0-b82e-621022196d32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:53:48.107323 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.107304    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c15d4442-c580-40c0-b82e-621022196d32" (UID: "c15d4442-c580-40c0-b82e-621022196d32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:53:48.107386 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.107346    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15d4442-c580-40c0-b82e-621022196d32-kube-api-access-bllsn" (OuterVolumeSpecName: "kube-api-access-bllsn") pod "c15d4442-c580-40c0-b82e-621022196d32" (UID: "c15d4442-c580-40c0-b82e-621022196d32"). InnerVolumeSpecName "kube-api-access-bllsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:53:48.206182 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.206145    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15d4442-c580-40c0-b82e-621022196d32-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:53:48.206182 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.206177    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllsn\" (UniqueName: \"kubernetes.io/projected/c15d4442-c580-40c0-b82e-621022196d32-kube-api-access-bllsn\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:53:48.206182 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.206187    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:53:48.206405 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.206196    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c15d4442-c580-40c0-b82e-621022196d32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:53:48.206405 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.206206    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c15d4442-c580-40c0-b82e-621022196d32-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:53:48.884199 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.884160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerStarted","Data":"98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466"}
Jul 07 11:53:48.884657 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.884218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerStarted","Data":"b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d"}
Jul 07 11:53:48.884657 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.884411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:48.885964 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.885939    2565 generic.go:356] "Generic (PLEG): container finished" podID="c15d4442-c580-40c0-b82e-621022196d32" containerID="8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5" exitCode=0
Jul 07 11:53:48.886082 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.885978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerDied","Data":"8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5"}
Jul 07 11:53:48.886082 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.885985    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"
Jul 07 11:53:48.886082 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.886029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257" event={"ID":"c15d4442-c580-40c0-b82e-621022196d32","Type":"ContainerDied","Data":"20497cd27cf8763e22c97f0deeb2cb5b2104e32af1e0ba9ef47ea0f410a8bad6"}
Jul 07 11:53:48.886082 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.886068    2565 scope.go:117] "RemoveContainer" containerID="ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e"
Jul 07 11:53:48.894201 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.894187    2565 scope.go:117] "RemoveContainer" containerID="8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5"
Jul 07 11:53:48.901307 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.901287    2565 scope.go:117] "RemoveContainer" containerID="dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594"
Jul 07 11:53:48.908178 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.908140    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podStartSLOduration=2.2788600949999998 podStartE2EDuration="5.908126906s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.629266811s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:53:48.906233374 +0000 UTC m=+813.607166235" watchObservedRunningTime="2026-07-07 11:53:48.908126906 +0000 UTC m=+813.609059765"
Jul 07 11:53:48.908239 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.908201    2565 scope.go:117] "RemoveContainer" containerID="ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e"
Jul 07 11:53:48.908463 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:53:48.908446    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e\": container with ID starting with ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e not found: ID does not exist" containerID="ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e"
Jul 07 11:53:48.908531 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.908470    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e"} err="failed to get container status \"ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e\": rpc error: code = NotFound desc = could not find container \"ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e\": container with ID starting with ca66db7c6c79a29d24ee3b5d7838996b40c97aa4cd3b7bd623e75120b146f30e not found: ID does not exist"
Jul 07 11:53:48.908531 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.908501    2565 scope.go:117] "RemoveContainer" containerID="8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5"
Jul 07 11:53:48.908756 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:53:48.908741    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5\": container with ID starting with 8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5 not found: ID does not exist" containerID="8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5"
Jul 07 11:53:48.908794 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.908759    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5"} err="failed to get container status \"8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5\": rpc error: code = NotFound desc = could not find container \"8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5\": container with ID starting with 8fb89d3e91d05c7ef1c01e0a6b41f9dd0a1ddc31bd30815422df7c48bb3ea2b5 not found: ID does not exist"
Jul 07 11:53:48.908794 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.908772    2565 scope.go:117] "RemoveContainer" containerID="dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594"
Jul 07 11:53:48.909000 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:53:48.908984    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594\": container with ID starting with dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594 not found: ID does not exist" containerID="dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594"
Jul 07 11:53:48.909047 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.909005    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594"} err="failed to get container status \"dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594\": rpc error: code = NotFound desc = could not find container \"dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594\": container with ID starting with dc9ed45b7e3558411b191a304ecc45de279edc4367dc3ec109cb488490ad1594 not found: ID does not exist"
Jul 07 11:53:48.920713 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.920692    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"]
Jul 07 11:53:48.926533 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:48.926513    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-fea16-predictor-7578c7bdf8-b4257"]
Jul 07 11:53:49.884466 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:49.884422    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15d4442-c580-40c0-b82e-621022196d32" path="/var/lib/kubelet/pods/c15d4442-c580-40c0-b82e-621022196d32/volumes"
Jul 07 11:53:49.890430 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:49.890406    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:49.891538 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:49.891514    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:53:50.894176 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:50.894137    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:53:52.835984 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:52.835949    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:55.898492 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:55.898454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:53:55.899031 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:55.898999    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:53:57.837114 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:57.837077    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:53:57.837461 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:53:57.837182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:54:02.837044 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:02.837005    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:05.899249 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:05.899211    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:54:07.836258 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:07.836223    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:12.837327 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:12.837285    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:54:13.271155 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.271131    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:54:13.393006 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.392921    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls\") pod \"926403a4-3d20-469a-b75f-cecaccfcdf03\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") "
Jul 07 11:54:13.393006 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.392967    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926403a4-3d20-469a-b75f-cecaccfcdf03-openshift-service-ca-bundle\") pod \"926403a4-3d20-469a-b75f-cecaccfcdf03\" (UID: \"926403a4-3d20-469a-b75f-cecaccfcdf03\") "
Jul 07 11:54:13.393348 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.393322    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926403a4-3d20-469a-b75f-cecaccfcdf03-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "926403a4-3d20-469a-b75f-cecaccfcdf03" (UID: "926403a4-3d20-469a-b75f-cecaccfcdf03"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:54:13.394972 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.394948    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "926403a4-3d20-469a-b75f-cecaccfcdf03" (UID: "926403a4-3d20-469a-b75f-cecaccfcdf03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:54:13.493669 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.493623    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/926403a4-3d20-469a-b75f-cecaccfcdf03-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:54:13.493669 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.493668    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/926403a4-3d20-469a-b75f-cecaccfcdf03-openshift-service-ca-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:54:13.973003 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.972966    2565 generic.go:356] "Generic (PLEG): container finished" podID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerID="3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c" exitCode=0
Jul 07 11:54:13.973470 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.973029    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"
Jul 07 11:54:13.973470 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.973050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" event={"ID":"926403a4-3d20-469a-b75f-cecaccfcdf03","Type":"ContainerDied","Data":"3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c"}
Jul 07 11:54:13.973470 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.973105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl" event={"ID":"926403a4-3d20-469a-b75f-cecaccfcdf03","Type":"ContainerDied","Data":"3f528acb200c7f64f88166a79613e7bed9519fc3f543af7ea821ff96f53738ba"}
Jul 07 11:54:13.973470 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.973127    2565 scope.go:117] "RemoveContainer" containerID="3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c"
Jul 07 11:54:13.980921 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.980906    2565 scope.go:117] "RemoveContainer" containerID="3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c"
Jul 07 11:54:13.981169 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:54:13.981150    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c\": container with ID starting with 3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c not found: ID does not exist" containerID="3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c"
Jul 07 11:54:13.981215 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.981178    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c"} err="failed to get container status \"3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c\": rpc error: code = NotFound desc = could not find container \"3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c\": container with ID starting with 3143271cb0d8dc58c24e3397b0abe75dc010154e60512fe39a11be9e41fb922c not found: ID does not exist"
Jul 07 11:54:13.989731 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.989710    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"]
Jul 07 11:54:13.994095 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:13.994072    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-fea16-7bbdc5b5df-wcghl"]
Jul 07 11:54:15.884716 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:15.884680    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" path="/var/lib/kubelet/pods/926403a4-3d20-469a-b75f-cecaccfcdf03/volumes"
Jul 07 11:54:15.899621 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:15.899591    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:54:25.899673 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:25.899639    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:54:35.899790 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:35.899746    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:54:45.899710 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:45.899674    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:54:55.899663 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:54:55.899623    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:55:05.900386 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:05.900358    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:55:19.321824 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:19.321797    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:55:19.323214 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:19.323191    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 11:55:23.349818 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.349784    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"]
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350043    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350054    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350064    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350069    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350078    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="storage-initializer"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350084    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="storage-initializer"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350096    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kube-rbac-proxy"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350101    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kube-rbac-proxy"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350147    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="926403a4-3d20-469a-b75f-cecaccfcdf03" containerName="model-chainer-raw-fea16"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350156    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kube-rbac-proxy"
Jul 07 11:55:23.350220 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.350163    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15d4442-c580-40c0-b82e-621022196d32" containerName="kserve-container"
Jul 07 11:55:23.353026 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.353007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:23.355557 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.355529    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e5dd1-kube-rbac-proxy-sar-config\""
Jul 07 11:55:23.355662 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.355541    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-e5dd1-serving-cert\""
Jul 07 11:55:23.363093 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.363073    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"]
Jul 07 11:55:23.421158 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.421130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:23.421158 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.421163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls\") pod \"model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:23.522513 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.522457    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:23.522513 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.522516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls\") pod \"model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:23.522755 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:55:23.522627    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-serving-cert: secret "model-chainer-raw-hpa-e5dd1-serving-cert" not found
Jul 07 11:55:23.522755 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:55:23.522684    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls podName:d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf nodeName:}" failed. No retries permitted until 2026-07-07 11:55:24.02266837 +0000 UTC m=+908.723601206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls") pod "model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" (UID: "d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf") : secret "model-chainer-raw-hpa-e5dd1-serving-cert" not found
Jul 07 11:55:23.523173 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:23.523149    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:24.027443 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:24.027411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls\") pod \"model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:24.029709 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:24.029692    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls\") pod \"model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:24.264020 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:24.263991    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:24.390727 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:24.390570    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"]
Jul 07 11:55:24.393616 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:55:24.393588    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a4bf02_86ab_4baf_b1ba_9d80c424b8bf.slice/crio-618171e1fc6dd3294be95f1a5da40d947f86e4bf47ca48477f4e00c3e1b1bb6a WatchSource:0}: Error finding container 618171e1fc6dd3294be95f1a5da40d947f86e4bf47ca48477f4e00c3e1b1bb6a: Status 404 returned error can't find the container with id 618171e1fc6dd3294be95f1a5da40d947f86e4bf47ca48477f4e00c3e1b1bb6a
Jul 07 11:55:25.226931 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:25.226892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" event={"ID":"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf","Type":"ContainerStarted","Data":"7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c"}
Jul 07 11:55:25.227130 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:25.226982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" event={"ID":"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf","Type":"ContainerStarted","Data":"618171e1fc6dd3294be95f1a5da40d947f86e4bf47ca48477f4e00c3e1b1bb6a"}
Jul 07 11:55:25.227130 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:25.227029    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:25.244012 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:25.243963    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podStartSLOduration=2.243947864 podStartE2EDuration="2.243947864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:55:25.24346544 +0000 UTC m=+909.944398298" watchObservedRunningTime="2026-07-07 11:55:25.243947864 +0000 UTC m=+909.944880722"
Jul 07 11:55:31.236475 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:31.236444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:33.391558 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.391524    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"]
Jul 07 11:55:33.392026 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.391739    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1" containerID="cri-o://7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c" gracePeriod=30
Jul 07 11:55:33.602009 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.601975    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"]
Jul 07 11:55:33.605840 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.605817    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.608521 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.608501    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-665a8-kube-rbac-proxy-sar-config\""
Jul 07 11:55:33.609649 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.609629    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-665a8-predictor-serving-cert\""
Jul 07 11:55:33.619188 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.619165    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"]
Jul 07 11:55:33.635562 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.635529    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"]
Jul 07 11:55:33.635828 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.635802    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" containerID="cri-o://b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d" gracePeriod=30
Jul 07 11:55:33.635916 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.635884    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kube-rbac-proxy" containerID="cri-o://98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466" gracePeriod=30
Jul 07 11:55:33.698573 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.698541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qskgn\" (UniqueName: \"kubernetes.io/projected/ac9374d3-c168-4169-b067-a62b00696c91-kube-api-access-qskgn\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.698573 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.698578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.698779 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.698600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9374d3-c168-4169-b067-a62b00696c91-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.799598 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.799560    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qskgn\" (UniqueName: \"kubernetes.io/projected/ac9374d3-c168-4169-b067-a62b00696c91-kube-api-access-qskgn\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.799598 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.799598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.799849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.799618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9374d3-c168-4169-b067-a62b00696c91-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.799849 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:55:33.799725    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-serving-cert: secret "message-dumper-raw-665a8-predictor-serving-cert" not found
Jul 07 11:55:33.799849 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:55:33.799793    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls podName:ac9374d3-c168-4169-b067-a62b00696c91 nodeName:}" failed. No retries permitted until 2026-07-07 11:55:34.299773626 +0000 UTC m=+919.000706480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls") pod "message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" (UID: "ac9374d3-c168-4169-b067-a62b00696c91") : secret "message-dumper-raw-665a8-predictor-serving-cert" not found
Jul 07 11:55:33.800240 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.800219    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9374d3-c168-4169-b067-a62b00696c91-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:33.809893 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:33.809862    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qskgn\" (UniqueName: \"kubernetes.io/projected/ac9374d3-c168-4169-b067-a62b00696c91-kube-api-access-qskgn\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:34.262165 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:34.262130    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerID="98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466" exitCode=2
Jul 07 11:55:34.262344 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:34.262203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerDied","Data":"98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466"}
Jul 07 11:55:34.303796 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:34.303752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:34.306107 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:34.306072    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls\") pod \"message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") " pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:34.517210 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:34.517108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:34.643739 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:34.643706    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"]
Jul 07 11:55:34.646764 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:55:34.646737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9374d3_c168_4169_b067_a62b00696c91.slice/crio-22823112f87f18f8c44ca5edb722d73ef0d6486575f071789cf19b986f59956d WatchSource:0}: Error finding container 22823112f87f18f8c44ca5edb722d73ef0d6486575f071789cf19b986f59956d: Status 404 returned error can't find the container with id 22823112f87f18f8c44ca5edb722d73ef0d6486575f071789cf19b986f59956d
Jul 07 11:55:35.266402 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:35.266366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" event={"ID":"ac9374d3-c168-4169-b067-a62b00696c91","Type":"ContainerStarted","Data":"22823112f87f18f8c44ca5edb722d73ef0d6486575f071789cf19b986f59956d"}
Jul 07 11:55:35.894765 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:35.894721    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 07 11:55:35.899312 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:35.899275    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 07 11:55:36.235619 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:36.235529    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:36.272192 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:36.272157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" event={"ID":"ac9374d3-c168-4169-b067-a62b00696c91","Type":"ContainerStarted","Data":"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"}
Jul 07 11:55:36.272382 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:36.272247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" event={"ID":"ac9374d3-c168-4169-b067-a62b00696c91","Type":"ContainerStarted","Data":"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"}
Jul 07 11:55:36.272382 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:36.272275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:36.294442 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:36.294381    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" podStartSLOduration=1.926832137 podStartE2EDuration="3.294364645s" totalImagesPullingTime="1.367532508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:55:36.292117812 +0000 UTC m=+920.993050671" watchObservedRunningTime="2026-07-07 11:55:36.294364645 +0000 UTC m=+920.995297503"
Jul 07 11:55:37.276314 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:37.276278    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:37.278153 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:37.278127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:38.580261 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.580239    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:55:38.639788 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.639713    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kserve-provision-location\") pod \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") "
Jul 07 11:55:38.639788 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.639756    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-proxy-tls\") pod \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") "
Jul 07 11:55:38.639971 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.639789    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-cabundle-cert\") pod \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") "
Jul 07 11:55:38.639971 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.639860    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-rbac-proxy-sar-config\") pod \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") "
Jul 07 11:55:38.640070 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.640005    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26rh9\" (UniqueName: \"kubernetes.io/projected/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-api-access-26rh9\") pod \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\" (UID: \"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8\") "
Jul 07 11:55:38.640169 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.640146    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" (UID: "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:55:38.640228 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.640192    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" (UID: "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:55:38.640228 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.640199    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" (UID: "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:55:38.640330 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.640247    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:55:38.640330 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.640260    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:55:38.640330 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.640269    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:55:38.641987 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.641960    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-api-access-26rh9" (OuterVolumeSpecName: "kube-api-access-26rh9") pod "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" (UID: "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8"). InnerVolumeSpecName "kube-api-access-26rh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:55:38.642092 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.641988    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" (UID: "b2935c9c-9c4c-4da4-9cf2-72d78d558ca8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:55:38.740769 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.740732    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:55:38.740769 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:38.740761    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rh9\" (UniqueName: \"kubernetes.io/projected/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8-kube-api-access-26rh9\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:55:39.284747 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.284711    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerID="b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d" exitCode=0
Jul 07 11:55:39.284924 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.284778    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"
Jul 07 11:55:39.284924 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.284790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerDied","Data":"b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d"}
Jul 07 11:55:39.284924 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.284853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc" event={"ID":"b2935c9c-9c4c-4da4-9cf2-72d78d558ca8","Type":"ContainerDied","Data":"a3364d81cf332b1a4a4b53842b7d833115b375655afdbeda768d2989c70fed04"}
Jul 07 11:55:39.284924 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.284879    2565 scope.go:117] "RemoveContainer" containerID="98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466"
Jul 07 11:55:39.293262 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.293243    2565 scope.go:117] "RemoveContainer" containerID="b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d"
Jul 07 11:55:39.300521 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.300501    2565 scope.go:117] "RemoveContainer" containerID="9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4"
Jul 07 11:55:39.307751 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.307730    2565 scope.go:117] "RemoveContainer" containerID="98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466"
Jul 07 11:55:39.307970 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:55:39.307951    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466\": container with ID starting with 98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466 not found: ID does not exist" containerID="98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466"
Jul 07 11:55:39.308036 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.307977    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466"} err="failed to get container status \"98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466\": rpc error: code = NotFound desc = could not find container \"98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466\": container with ID starting with 98220cc0bb95d729be1b4edaac972d72ee41dd2861d575355da446949a7aa466 not found: ID does not exist"
Jul 07 11:55:39.308036 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.307996    2565 scope.go:117] "RemoveContainer" containerID="b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d"
Jul 07 11:55:39.308247 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:55:39.308227    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d\": container with ID starting with b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d not found: ID does not exist" containerID="b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d"
Jul 07 11:55:39.308307 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.308252    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d"} err="failed to get container status \"b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d\": rpc error: code = NotFound desc = could not find container \"b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d\": container with ID starting with b759ed99580cfb1169fec540097ce767ba63ce76ce80e8bb9b27083a49f4534d not found: ID does not exist"
Jul 07 11:55:39.308307 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.308268    2565 scope.go:117] "RemoveContainer" containerID="9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4"
Jul 07 11:55:39.308978 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:55:39.308957    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4\": container with ID starting with 9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4 not found: ID does not exist" containerID="9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4"
Jul 07 11:55:39.309066 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.308980    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4"} err="failed to get container status \"9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4\": rpc error: code = NotFound desc = could not find container \"9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4\": container with ID starting with 9567dcf67ee65c046c515aee4991c1cb2e1ce45dca3baa62742bb47434e42bd4 not found: ID does not exist"
Jul 07 11:55:39.309193 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.309175    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"]
Jul 07 11:55:39.312671 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.312649    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-e5dd1-predictor-7bf85b4b4b-zd4wc"]
Jul 07 11:55:39.884212 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:39.884175    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" path="/var/lib/kubelet/pods/b2935c9c-9c4c-4da4-9cf2-72d78d558ca8/volumes"
Jul 07 11:55:41.235389 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:41.235345    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:44.289606 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:44.289574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:55:46.235709 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:46.235675    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:46.236101 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:46.235780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:55:51.235262 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:51.235223    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:53.639762 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.639726    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"]
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="storage-initializer"
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640027    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="storage-initializer"
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640040    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kube-rbac-proxy"
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kube-rbac-proxy"
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640054    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container"
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640059    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container"
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640101    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kserve-container"
Jul 07 11:55:53.640145 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.640108    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2935c9c-9c4c-4da4-9cf2-72d78d558ca8" containerName="kube-rbac-proxy"
Jul 07 11:55:53.644921 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.644896    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.647899 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.647876    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 11:55:53.648050 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.648011    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-665a8-predictor-serving-cert\""
Jul 07 11:55:53.648050 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.648026    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 11:55:53.648174 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.648028    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-665a8-kube-rbac-proxy-sar-config\""
Jul 07 11:55:53.655932 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.655906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"]
Jul 07 11:55:53.761322 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.761287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.761560 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.761337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6b2\" (UniqueName: \"kubernetes.io/projected/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-api-access-pl6b2\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.761560 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.761360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-cabundle-cert\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.761560 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.761379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b9f9f9-462a-4322-8b53-40b9d8753383-kserve-provision-location\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.761560 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.761395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b9f9f9-462a-4322-8b53-40b9d8753383-proxy-tls\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.862530 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.862466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.862707 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.862543    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6b2\" (UniqueName: \"kubernetes.io/projected/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-api-access-pl6b2\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.862707 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.862565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-cabundle-cert\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.862707 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.862580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b9f9f9-462a-4322-8b53-40b9d8753383-kserve-provision-location\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.862707 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.862596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b9f9f9-462a-4322-8b53-40b9d8753383-proxy-tls\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.863008 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.862983    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b9f9f9-462a-4322-8b53-40b9d8753383-kserve-provision-location\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.863213 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.863195    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.863258 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.863212    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-cabundle-cert\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.865005 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.864987    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b9f9f9-462a-4322-8b53-40b9d8753383-proxy-tls\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.871415 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.871391    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6b2\" (UniqueName: \"kubernetes.io/projected/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-api-access-pl6b2\") pod \"isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") " pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:53.956100 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:53.955996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:54.091936 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:54.091902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"]
Jul 07 11:55:54.094782 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:55:54.094751    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b9f9f9_462a_4322_8b53_40b9d8753383.slice/crio-e7dd9a92e8d96eac3a0e4ccc290265579684b5e6a1f71ad32915f899a7a22b7c WatchSource:0}: Error finding container e7dd9a92e8d96eac3a0e4ccc290265579684b5e6a1f71ad32915f899a7a22b7c: Status 404 returned error can't find the container with id e7dd9a92e8d96eac3a0e4ccc290265579684b5e6a1f71ad32915f899a7a22b7c
Jul 07 11:55:54.340080 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:54.340043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerStarted","Data":"fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961"}
Jul 07 11:55:54.340255 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:54.340125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerStarted","Data":"e7dd9a92e8d96eac3a0e4ccc290265579684b5e6a1f71ad32915f899a7a22b7c"}
Jul 07 11:55:56.234872 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:56.234836    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:55:58.355213 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:58.355180    2565 generic.go:356] "Generic (PLEG): container finished" podID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerID="fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961" exitCode=0
Jul 07 11:55:58.355669 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:58.355228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerDied","Data":"fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961"}
Jul 07 11:55:59.361898 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:59.361862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerStarted","Data":"d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9"}
Jul 07 11:55:59.362373 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:59.361936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerStarted","Data":"df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54"}
Jul 07 11:55:59.362373 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:59.361974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerStarted","Data":"440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c"}
Jul 07 11:55:59.362373 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:59.362213    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:55:59.386956 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:55:59.386905    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podStartSLOduration=2.6037222939999998 podStartE2EDuration="6.386886586s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783164292s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:55:59.384816836 +0000 UTC m=+944.085749686" watchObservedRunningTime="2026-07-07 11:55:59.386886586 +0000 UTC m=+944.087819447"
Jul 07 11:56:00.365327 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:00.365289    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:56:00.365704 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:00.365358    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:56:00.366705 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:00.366679    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:00.367344 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:00.367323    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:01.235806 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:01.235763    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:01.369345 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:01.369297    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:01.369814 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:01.369745    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:03.545518 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:03.545493    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:56:03.637130 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:03.637084    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-openshift-service-ca-bundle\") pod \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") "
Jul 07 11:56:03.637336 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:03.637186    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls\") pod \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\" (UID: \"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf\") "
Jul 07 11:56:03.637443 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:03.637417    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" (UID: "d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:56:03.639094 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:03.639070    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" (UID: "d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:56:03.738696 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:03.738602    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:56:03.738696 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:03.738628    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf-openshift-service-ca-bundle\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:56:04.381583 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.381542    2565 generic.go:356] "Generic (PLEG): container finished" podID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerID="7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c" exitCode=0
Jul 07 11:56:04.381774 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.381604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"
Jul 07 11:56:04.381774 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.381633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" event={"ID":"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf","Type":"ContainerDied","Data":"7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c"}
Jul 07 11:56:04.381774 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.381690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk" event={"ID":"d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf","Type":"ContainerDied","Data":"618171e1fc6dd3294be95f1a5da40d947f86e4bf47ca48477f4e00c3e1b1bb6a"}
Jul 07 11:56:04.381774 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.381717    2565 scope.go:117] "RemoveContainer" containerID="7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c"
Jul 07 11:56:04.389460 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.389444    2565 scope.go:117] "RemoveContainer" containerID="7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c"
Jul 07 11:56:04.389727 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:56:04.389706    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c\": container with ID starting with 7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c not found: ID does not exist" containerID="7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c"
Jul 07 11:56:04.389796 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.389739    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c"} err="failed to get container status \"7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c\": rpc error: code = NotFound desc = could not find container \"7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c\": container with ID starting with 7ddd50a17dcf1f6596d6bf63a3dd1c6d9f5c7ed2e078a97af77b4888180b3b1c not found: ID does not exist"
Jul 07 11:56:04.397056 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.397037    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"]
Jul 07 11:56:04.402204 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:04.402186    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-e5dd1-767dd98897-6vnrk"]
Jul 07 11:56:05.884315 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:05.884284    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" path="/var/lib/kubelet/pods/d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf/volumes"
Jul 07 11:56:06.374063 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:06.374035    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:56:06.374669 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:06.374639    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:06.375009 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:06.374987    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:16.375451 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:16.375409    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:16.375895 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:16.375871    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:26.374887 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:26.374851    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:26.375408 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:26.375330    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:36.374807 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:36.374713    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:36.375269 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:36.375230    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:46.374844 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:46.374803    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:46.375292 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:46.375215    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:56:56.374533 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:56.374470    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:56:56.375000 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:56:56.374926    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:57:06.374762 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:06.374713    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:57:06.375198 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:06.375075    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:57:11.884927 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:11.884900    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:57:11.885304 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:11.884978    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:57:18.649859 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.649828    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw_ac9374d3-c168-4169-b067-a62b00696c91/kserve-container/0.log"
Jul 07 11:57:18.834792 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.834757    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"]
Jul 07 11:57:18.835244 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.835071    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" containerID="cri-o://440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c" gracePeriod=30
Jul 07 11:57:18.835244 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.835134    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy" containerID="cri-o://df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54" gracePeriod=30
Jul 07 11:57:18.835989 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.835536    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" containerID="cri-o://d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9" gracePeriod=30
Jul 07 11:57:18.898901 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.898861    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"]
Jul 07 11:57:18.899148 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.899134    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1"
Jul 07 11:57:18.899206 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.899148    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1"
Jul 07 11:57:18.899257 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.899208    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a4bf02-86ab-4baf-b1ba-9d80c424b8bf" containerName="model-chainer-raw-hpa-e5dd1"
Jul 07 11:57:18.902346 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.902290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:18.905093 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.905068    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-8255d-kube-rbac-proxy-sar-config\""
Jul 07 11:57:18.905226 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.905076    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-8255d-predictor-serving-cert\""
Jul 07 11:57:18.913229 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.913204    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"]
Jul 07 11:57:18.967240 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.967208    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"]
Jul 07 11:57:18.967578 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.967515    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kserve-container" containerID="cri-o://f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c" gracePeriod=30
Jul 07 11:57:18.967578 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:18.967556    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kube-rbac-proxy" containerID="cri-o://cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19" gracePeriod=30
Jul 07 11:57:19.026985 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.026947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.027147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.027017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlbz\" (UniqueName: \"kubernetes.io/projected/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-api-access-5xlbz\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.027147 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.027043    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.027252 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.027159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36720b6c-40c2-4611-a629-bf30ea60c21e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.027252 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.027218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36720b6c-40c2-4611-a629-bf30ea60c21e-proxy-tls\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.128536 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.128505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36720b6c-40c2-4611-a629-bf30ea60c21e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.128720 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.128542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36720b6c-40c2-4611-a629-bf30ea60c21e-proxy-tls\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.128720 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.128591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.128720 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.128623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlbz\" (UniqueName: \"kubernetes.io/projected/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-api-access-5xlbz\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.128720 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.128640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.128993 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.128970    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36720b6c-40c2-4611-a629-bf30ea60c21e-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.129374 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.129352    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.129374 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.129359    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-cabundle-cert\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.131066 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.131045    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36720b6c-40c2-4611-a629-bf30ea60c21e-proxy-tls\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.138183 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.138163    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlbz\" (UniqueName: \"kubernetes.io/projected/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-api-access-5xlbz\") pod \"isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.213208 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.213186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:19.215293 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.215274    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:57:19.330029 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.330000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls\") pod \"ac9374d3-c168-4169-b067-a62b00696c91\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") "
Jul 07 11:57:19.330257 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.330052    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9374d3-c168-4169-b067-a62b00696c91-kube-rbac-proxy-sar-config\") pod \"ac9374d3-c168-4169-b067-a62b00696c91\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") "
Jul 07 11:57:19.330257 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.330163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qskgn\" (UniqueName: \"kubernetes.io/projected/ac9374d3-c168-4169-b067-a62b00696c91-kube-api-access-qskgn\") pod \"ac9374d3-c168-4169-b067-a62b00696c91\" (UID: \"ac9374d3-c168-4169-b067-a62b00696c91\") "
Jul 07 11:57:19.330425 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.330401    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9374d3-c168-4169-b067-a62b00696c91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ac9374d3-c168-4169-b067-a62b00696c91" (UID: "ac9374d3-c168-4169-b067-a62b00696c91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:57:19.332014 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.331990    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac9374d3-c168-4169-b067-a62b00696c91" (UID: "ac9374d3-c168-4169-b067-a62b00696c91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:57:19.332183 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.332159    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9374d3-c168-4169-b067-a62b00696c91-kube-api-access-qskgn" (OuterVolumeSpecName: "kube-api-access-qskgn") pod "ac9374d3-c168-4169-b067-a62b00696c91" (UID: "ac9374d3-c168-4169-b067-a62b00696c91"). InnerVolumeSpecName "kube-api-access-qskgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:57:19.350152 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.350126    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"]
Jul 07 11:57:19.352859 ip-10-0-143-227 kubenswrapper[2565]: W0707 11:57:19.352838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36720b6c_40c2_4611_a629_bf30ea60c21e.slice/crio-7882b2a27361cde7796a45d4f7df2abd06cb332159b535a523450cd2ce917ca3 WatchSource:0}: Error finding container 7882b2a27361cde7796a45d4f7df2abd06cb332159b535a523450cd2ce917ca3: Status 404 returned error can't find the container with id 7882b2a27361cde7796a45d4f7df2abd06cb332159b535a523450cd2ce917ca3
Jul 07 11:57:19.431110 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.431086    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac9374d3-c168-4169-b067-a62b00696c91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:19.431110 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.431110    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qskgn\" (UniqueName: \"kubernetes.io/projected/ac9374d3-c168-4169-b067-a62b00696c91-kube-api-access-qskgn\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:19.431254 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.431127    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac9374d3-c168-4169-b067-a62b00696c91-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:19.663326 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.663292    2565 generic.go:356] "Generic (PLEG): container finished" podID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerID="df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54" exitCode=2
Jul 07 11:57:19.663784 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.663365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerDied","Data":"df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54"}
Jul 07 11:57:19.665055 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.665029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerStarted","Data":"bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1"}
Jul 07 11:57:19.665182 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.665083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerStarted","Data":"7882b2a27361cde7796a45d4f7df2abd06cb332159b535a523450cd2ce917ca3"}
Jul 07 11:57:19.666564 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.666541    2565 generic.go:356] "Generic (PLEG): container finished" podID="ac9374d3-c168-4169-b067-a62b00696c91" containerID="cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19" exitCode=2
Jul 07 11:57:19.666564 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.666561    2565 generic.go:356] "Generic (PLEG): container finished" podID="ac9374d3-c168-4169-b067-a62b00696c91" containerID="f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c" exitCode=2
Jul 07 11:57:19.666686 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.666580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" event={"ID":"ac9374d3-c168-4169-b067-a62b00696c91","Type":"ContainerDied","Data":"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"}
Jul 07 11:57:19.666686 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.666605    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"
Jul 07 11:57:19.666686 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.666622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" event={"ID":"ac9374d3-c168-4169-b067-a62b00696c91","Type":"ContainerDied","Data":"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"}
Jul 07 11:57:19.666686 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.666646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw" event={"ID":"ac9374d3-c168-4169-b067-a62b00696c91","Type":"ContainerDied","Data":"22823112f87f18f8c44ca5edb722d73ef0d6486575f071789cf19b986f59956d"}
Jul 07 11:57:19.666686 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.666672    2565 scope.go:117] "RemoveContainer" containerID="cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"
Jul 07 11:57:19.674620 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.674597    2565 scope.go:117] "RemoveContainer" containerID="f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"
Jul 07 11:57:19.682605 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.682577    2565 scope.go:117] "RemoveContainer" containerID="cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"
Jul 07 11:57:19.682904 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:57:19.682883    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19\": container with ID starting with cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19 not found: ID does not exist" containerID="cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"
Jul 07 11:57:19.682991 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.682916    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"} err="failed to get container status \"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19\": rpc error: code = NotFound desc = could not find container \"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19\": container with ID starting with cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19 not found: ID does not exist"
Jul 07 11:57:19.682991 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.682944    2565 scope.go:117] "RemoveContainer" containerID="f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"
Jul 07 11:57:19.683175 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:57:19.683159    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c\": container with ID starting with f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c not found: ID does not exist" containerID="f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"
Jul 07 11:57:19.683218 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.683181    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"} err="failed to get container status \"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c\": rpc error: code = NotFound desc = could not find container \"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c\": container with ID starting with f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c not found: ID does not exist"
Jul 07 11:57:19.683218 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.683197    2565 scope.go:117] "RemoveContainer" containerID="cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"
Jul 07 11:57:19.683455 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.683431    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19"} err="failed to get container status \"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19\": rpc error: code = NotFound desc = could not find container \"cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19\": container with ID starting with cd91d2998bed17d8b3ee32448826cf2f79d37bedf33f076db1362c94274bfc19 not found: ID does not exist"
Jul 07 11:57:19.683520 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.683459    2565 scope.go:117] "RemoveContainer" containerID="f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"
Jul 07 11:57:19.683740 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.683719    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c"} err="failed to get container status \"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c\": rpc error: code = NotFound desc = could not find container \"f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c\": container with ID starting with f6789a898ea377f8447db3a14b95c9d166d3446f207861a67835bbb9ef29479c not found: ID does not exist"
Jul 07 11:57:19.699623 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.699580    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"]
Jul 07 11:57:19.711929 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.711857    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-665a8-predictor-58bc9f4578-lc7bw"]
Jul 07 11:57:19.884658 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:19.884619    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac9374d3-c168-4169-b067-a62b00696c91" path="/var/lib/kubelet/pods/ac9374d3-c168-4169-b067-a62b00696c91/volumes"
Jul 07 11:57:21.370587 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:21.370543    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 11:57:21.881426 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:21.881384    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:57:21.881770 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:21.881742    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:57:23.683946 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:23.683857    2565 generic.go:356] "Generic (PLEG): container finished" podID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerID="440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c" exitCode=0
Jul 07 11:57:23.683946 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:23.683930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerDied","Data":"440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c"}
Jul 07 11:57:23.685264 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:23.685245    2565 generic.go:356] "Generic (PLEG): container finished" podID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerID="bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1" exitCode=0
Jul 07 11:57:23.685356 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:23.685311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerDied","Data":"bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1"}
Jul 07 11:57:24.691065 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:24.691032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerStarted","Data":"d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584"}
Jul 07 11:57:24.691433 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:24.691092    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerStarted","Data":"a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf"}
Jul 07 11:57:24.691433 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:24.691373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:24.691552 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:24.691537    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:24.692862 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:24.692838    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:57:24.712466 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:24.712423    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podStartSLOduration=3.045514154 podStartE2EDuration="6.712409828s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.666895674s" isStatefulPod=false podCreationTimestamp="2026-07-07 11:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 11:57:24.711139068 +0000 UTC m=+1029.412071954" watchObservedRunningTime="2026-07-07 11:57:24.712409828 +0000 UTC m=+1029.413342686"
Jul 07 11:57:25.694747 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:25.694705    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:57:26.369546 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:26.369506    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 11:57:30.699032 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:30.699002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 11:57:30.699543 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:30.699441    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:57:31.369899 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:31.369857    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 11:57:31.370083 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:31.370010    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:57:31.880833 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:31.880788    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:57:31.881208 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:31.881096    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:57:36.370196 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:36.370150    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 11:57:40.699564 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:40.699524    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:57:41.369630 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:41.369588    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 11:57:41.881403 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:41.881362    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 07 11:57:41.881855 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:41.881699    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 11:57:41.884586 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:41.884566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:57:41.884722 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:41.884639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:57:46.369858 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:46.369807    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 07 11:57:49.014570 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.014542    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:57:49.060333 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060303    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-cabundle-cert\") pod \"81b9f9f9-462a-4322-8b53-40b9d8753383\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") "
Jul 07 11:57:49.060523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060352    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-rbac-proxy-sar-config\") pod \"81b9f9f9-462a-4322-8b53-40b9d8753383\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") "
Jul 07 11:57:49.060523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060370    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl6b2\" (UniqueName: \"kubernetes.io/projected/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-api-access-pl6b2\") pod \"81b9f9f9-462a-4322-8b53-40b9d8753383\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") "
Jul 07 11:57:49.060523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b9f9f9-462a-4322-8b53-40b9d8753383-proxy-tls\") pod \"81b9f9f9-462a-4322-8b53-40b9d8753383\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") "
Jul 07 11:57:49.060523 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060427    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b9f9f9-462a-4322-8b53-40b9d8753383-kserve-provision-location\") pod \"81b9f9f9-462a-4322-8b53-40b9d8753383\" (UID: \"81b9f9f9-462a-4322-8b53-40b9d8753383\") "
Jul 07 11:57:49.060849 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060823    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81b9f9f9-462a-4322-8b53-40b9d8753383" (UID: "81b9f9f9-462a-4322-8b53-40b9d8753383"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:57:49.060928 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060828    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81b9f9f9-462a-4322-8b53-40b9d8753383" (UID: "81b9f9f9-462a-4322-8b53-40b9d8753383"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 11:57:49.060928 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.060845    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b9f9f9-462a-4322-8b53-40b9d8753383-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81b9f9f9-462a-4322-8b53-40b9d8753383" (UID: "81b9f9f9-462a-4322-8b53-40b9d8753383"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 11:57:49.062546 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.062519    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b9f9f9-462a-4322-8b53-40b9d8753383-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81b9f9f9-462a-4322-8b53-40b9d8753383" (UID: "81b9f9f9-462a-4322-8b53-40b9d8753383"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 11:57:49.062594 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.062552    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-api-access-pl6b2" (OuterVolumeSpecName: "kube-api-access-pl6b2") pod "81b9f9f9-462a-4322-8b53-40b9d8753383" (UID: "81b9f9f9-462a-4322-8b53-40b9d8753383"). InnerVolumeSpecName "kube-api-access-pl6b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 11:57:49.161930 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.161845    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81b9f9f9-462a-4322-8b53-40b9d8753383-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:49.161930 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.161877    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:49.161930 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.161888    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:49.161930 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.161897    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl6b2\" (UniqueName: \"kubernetes.io/projected/81b9f9f9-462a-4322-8b53-40b9d8753383-kube-api-access-pl6b2\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:49.161930 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.161906    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81b9f9f9-462a-4322-8b53-40b9d8753383-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 11:57:49.782826 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.782792    2565 generic.go:356] "Generic (PLEG): container finished" podID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerID="d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9" exitCode=137
Jul 07 11:57:49.783016 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.782850    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"
Jul 07 11:57:49.783016 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.782854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerDied","Data":"d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9"}
Jul 07 11:57:49.783016 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.782924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx" event={"ID":"81b9f9f9-462a-4322-8b53-40b9d8753383","Type":"ContainerDied","Data":"e7dd9a92e8d96eac3a0e4ccc290265579684b5e6a1f71ad32915f899a7a22b7c"}
Jul 07 11:57:49.783016 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.782967    2565 scope.go:117] "RemoveContainer" containerID="d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9"
Jul 07 11:57:49.791159 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.791144    2565 scope.go:117] "RemoveContainer" containerID="df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54"
Jul 07 11:57:49.798568 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.798547    2565 scope.go:117] "RemoveContainer" containerID="440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c"
Jul 07 11:57:49.807295 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.807267    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"]
Jul 07 11:57:49.808034 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.808013    2565 scope.go:117] "RemoveContainer" containerID="fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961"
Jul 07 11:57:49.810438 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.810420    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-665a8-predictor-57894bfdbf-s5jgx"]
Jul 07 11:57:49.815780 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.815763    2565 scope.go:117] "RemoveContainer" containerID="d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9"
Jul 07 11:57:49.816044 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:57:49.816017    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9\": container with ID starting with d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9 not found: ID does not exist" containerID="d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9"
Jul 07 11:57:49.816118 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.816057    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9"} err="failed to get container status \"d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9\": rpc error: code = NotFound desc = could not find container \"d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9\": container with ID starting with d482314c4198405a2442aaf5302eb773b26ed0787696d2839cd8b63d3444d3d9 not found: ID does not exist"
Jul 07 11:57:49.816118 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.816085    2565 scope.go:117] "RemoveContainer" containerID="df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54"
Jul 07 11:57:49.816323 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:57:49.816306    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54\": container with ID starting with df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54 not found: ID does not exist" containerID="df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54"
Jul 07 11:57:49.816367 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.816329    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54"} err="failed to get container status \"df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54\": rpc error: code = NotFound desc = could not find container \"df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54\": container with ID starting with df84a2cb896963eea2f0a2a76dc4b2c01e0cba7088f6ec86d3b1cd2c5a03dd54 not found: ID does not exist"
Jul 07 11:57:49.816367 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.816346    2565 scope.go:117] "RemoveContainer" containerID="440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c"
Jul 07 11:57:49.816621 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:57:49.816602    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c\": container with ID starting with 440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c not found: ID does not exist" containerID="440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c"
Jul 07 11:57:49.816688 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.816636    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c"} err="failed to get container status \"440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c\": rpc error: code = NotFound desc = could not find container \"440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c\": container with ID starting with 440f5e3b81d1d4536666304eb920b17a87120915b021853ae7348c2103ba3a3c not found: ID does not exist"
Jul 07 11:57:49.816688 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.816656    2565 scope.go:117] "RemoveContainer" containerID="fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961"
Jul 07 11:57:49.816894 ip-10-0-143-227 kubenswrapper[2565]: E0707 11:57:49.816877    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961\": container with ID starting with fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961 not found: ID does not exist" containerID="fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961"
Jul 07 11:57:49.816971 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.816898    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961"} err="failed to get container status \"fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961\": rpc error: code = NotFound desc = could not find container \"fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961\": container with ID starting with fc6c8774aa051d3ab4a3b3aebe248269c993fcabae6b5fbefef9d250965fc961 not found: ID does not exist"
Jul 07 11:57:49.885044 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:49.885011    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" path="/var/lib/kubelet/pods/81b9f9f9-462a-4322-8b53-40b9d8753383/volumes"
Jul 07 11:57:50.700117 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:57:50.700085    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:58:00.699628 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:58:00.699591    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:58:10.699995 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:58:10.699913    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:58:20.700058 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:58:20.700019    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:58:30.699991 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:58:30.699954    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:58:40.700350 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:58:40.700315    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:58:50.699612 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:58:50.699577    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:59:00.699918 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:59:00.699882    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:59:00.880125 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:59:00.880088    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:59:10.880125 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:59:10.880084    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:59:20.880546 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:59:20.880469    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:59:30.881042 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:59:30.880993    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:59:40.881065 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:59:40.880978    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 11:59:50.880059 ip-10-0-143-227 kubenswrapper[2565]: I0707 11:59:50.880013    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 12:00:00.881523 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:00.881467    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 12:00:09.085572 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:09.085541    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"]
Jul 07 12:00:09.086079 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:09.085788    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" containerID="cri-o://a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf" gracePeriod=30
Jul 07 12:00:09.086079 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:09.085852    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kube-rbac-proxy" containerID="cri-o://d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584" gracePeriod=30
Jul 07 12:00:09.263618 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:09.263577    2565 generic.go:356] "Generic (PLEG): container finished" podID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerID="d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584" exitCode=2
Jul 07 12:00:09.263785 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:09.263656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerDied","Data":"d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584"}
Jul 07 12:00:10.695884 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:10.695835    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 12:00:10.880728 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:10.880688    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 07 12:00:14.232990 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.232940    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"]
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233195    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kserve-container"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233206    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kserve-container"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233214    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kube-rbac-proxy"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233219    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kube-rbac-proxy"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233230    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233236    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233244    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233249    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233260    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233265    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233274    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="storage-initializer"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233279    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="storage-initializer"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233321    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kube-rbac-proxy"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233327    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kserve-container"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233334    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="kube-rbac-proxy"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233339    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b9f9f9-462a-4322-8b53-40b9d8753383" containerName="agent"
Jul 07 12:00:14.233385 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.233345    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9374d3-c168-4169-b067-a62b00696c91" containerName="kserve-container"
Jul 07 12:00:14.236159 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.236141    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.238826 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.238801    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2abe02-predictor-serving-cert\""
Jul 07 12:00:14.238943 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.238831    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-2abe02-kube-rbac-proxy-sar-config\""
Jul 07 12:00:14.248344 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.248317    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"]
Jul 07 12:00:14.430734 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.430697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5904b2c6-0388-44b0-bc40-b72a45ffa369-kserve-provision-location\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.430920 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.430741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5904b2c6-0388-44b0-bc40-b72a45ffa369-proxy-tls\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.430920 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.430776    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.430920 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.430800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bxq\" (UniqueName: \"kubernetes.io/projected/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-api-access-n4bxq\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.430920 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.430828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-cabundle-cert\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.531771 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.531675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5904b2c6-0388-44b0-bc40-b72a45ffa369-kserve-provision-location\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.531771 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.531719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5904b2c6-0388-44b0-bc40-b72a45ffa369-proxy-tls\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.531771 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.531740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.531771 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.531758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bxq\" (UniqueName: \"kubernetes.io/projected/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-api-access-n4bxq\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.532090 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.531783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-cabundle-cert\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.532208 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.532188    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5904b2c6-0388-44b0-bc40-b72a45ffa369-kserve-provision-location\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.532405 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.532385    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-rbac-proxy-sar-config\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.532530 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.532507    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-cabundle-cert\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.534211 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.534193    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5904b2c6-0388-44b0-bc40-b72a45ffa369-proxy-tls\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.540810 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.540789    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bxq\" (UniqueName: \"kubernetes.io/projected/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-api-access-n4bxq\") pod \"isvc-primary-2abe02-predictor-66db8475c5-29lx8\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") " pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.546619 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.546601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:14.675786 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:14.675612    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"]
Jul 07 12:00:14.678520 ip-10-0-143-227 kubenswrapper[2565]: W0707 12:00:14.678464    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5904b2c6_0388_44b0_bc40_b72a45ffa369.slice/crio-43b0bbe96b1a1428f6a3609166188a0b6e8c48a2e19e524624277204d52efb02 WatchSource:0}: Error finding container 43b0bbe96b1a1428f6a3609166188a0b6e8c48a2e19e524624277204d52efb02: Status 404 returned error can't find the container with id 43b0bbe96b1a1428f6a3609166188a0b6e8c48a2e19e524624277204d52efb02
Jul 07 12:00:15.287152 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:15.287115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerStarted","Data":"7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b"}
Jul 07 12:00:15.287603 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:15.287181    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerStarted","Data":"43b0bbe96b1a1428f6a3609166188a0b6e8c48a2e19e524624277204d52efb02"}
Jul 07 12:00:15.695347 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:15.695311    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 07 12:00:18.928283 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.928260    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 12:00:18.967266 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967235    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-rbac-proxy-sar-config\") pod \"36720b6c-40c2-4611-a629-bf30ea60c21e\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") "
Jul 07 12:00:18.967434 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967279    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xlbz\" (UniqueName: \"kubernetes.io/projected/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-api-access-5xlbz\") pod \"36720b6c-40c2-4611-a629-bf30ea60c21e\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") "
Jul 07 12:00:18.967434 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36720b6c-40c2-4611-a629-bf30ea60c21e-kserve-provision-location\") pod \"36720b6c-40c2-4611-a629-bf30ea60c21e\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") "
Jul 07 12:00:18.967434 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967349    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36720b6c-40c2-4611-a629-bf30ea60c21e-proxy-tls\") pod \"36720b6c-40c2-4611-a629-bf30ea60c21e\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") "
Jul 07 12:00:18.967434 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967377    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-cabundle-cert\") pod \"36720b6c-40c2-4611-a629-bf30ea60c21e\" (UID: \"36720b6c-40c2-4611-a629-bf30ea60c21e\") "
Jul 07 12:00:18.967825 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967674    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36720b6c-40c2-4611-a629-bf30ea60c21e" (UID: "36720b6c-40c2-4611-a629-bf30ea60c21e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:00:18.967825 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967778    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36720b6c-40c2-4611-a629-bf30ea60c21e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36720b6c-40c2-4611-a629-bf30ea60c21e" (UID: "36720b6c-40c2-4611-a629-bf30ea60c21e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:00:18.968004 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.967836    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36720b6c-40c2-4611-a629-bf30ea60c21e" (UID: "36720b6c-40c2-4611-a629-bf30ea60c21e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:00:18.969763 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.969727    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-api-access-5xlbz" (OuterVolumeSpecName: "kube-api-access-5xlbz") pod "36720b6c-40c2-4611-a629-bf30ea60c21e" (UID: "36720b6c-40c2-4611-a629-bf30ea60c21e"). InnerVolumeSpecName "kube-api-access-5xlbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:00:18.969852 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:18.969809    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36720b6c-40c2-4611-a629-bf30ea60c21e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36720b6c-40c2-4611-a629-bf30ea60c21e" (UID: "36720b6c-40c2-4611-a629-bf30ea60c21e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:00:19.068179 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.068096    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36720b6c-40c2-4611-a629-bf30ea60c21e-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:00:19.068179 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.068125    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36720b6c-40c2-4611-a629-bf30ea60c21e-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:00:19.068179 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.068135    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:00:19.068179 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.068143    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:00:19.068179 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.068152    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlbz\" (UniqueName: \"kubernetes.io/projected/36720b6c-40c2-4611-a629-bf30ea60c21e-kube-api-access-5xlbz\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:00:19.301873 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.301839    2565 generic.go:356] "Generic (PLEG): container finished" podID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerID="7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b" exitCode=0
Jul 07 12:00:19.302031 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.301917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerDied","Data":"7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b"}
Jul 07 12:00:19.303777 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.303749    2565 generic.go:356] "Generic (PLEG): container finished" podID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerID="a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf" exitCode=0
Jul 07 12:00:19.303860 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.303806    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"
Jul 07 12:00:19.303928 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.303808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerDied","Data":"a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf"}
Jul 07 12:00:19.303994 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.303944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78" event={"ID":"36720b6c-40c2-4611-a629-bf30ea60c21e","Type":"ContainerDied","Data":"7882b2a27361cde7796a45d4f7df2abd06cb332159b535a523450cd2ce917ca3"}
Jul 07 12:00:19.304047 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.303990    2565 scope.go:117] "RemoveContainer" containerID="d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584"
Jul 07 12:00:19.314620 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.314604    2565 scope.go:117] "RemoveContainer" containerID="a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf"
Jul 07 12:00:19.322111 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.322092    2565 scope.go:117] "RemoveContainer" containerID="bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1"
Jul 07 12:00:19.338536 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.338173    2565 scope.go:117] "RemoveContainer" containerID="d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584"
Jul 07 12:00:19.338937 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.338893    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"]
Jul 07 12:00:19.341766 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:00:19.341593    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584\": container with ID starting with d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584 not found: ID does not exist" containerID="d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584"
Jul 07 12:00:19.341766 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.341631    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584"} err="failed to get container status \"d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584\": rpc error: code = NotFound desc = could not find container \"d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584\": container with ID starting with d83be9eb27055c34cbbd4f1e3acc45dcd09d34e3b9751f5e5ecba6a72170d584 not found: ID does not exist"
Jul 07 12:00:19.341766 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.341658    2565 scope.go:117] "RemoveContainer" containerID="a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf"
Jul 07 12:00:19.343214 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:00:19.342344    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf\": container with ID starting with a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf not found: ID does not exist" containerID="a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf"
Jul 07 12:00:19.343214 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.342397    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf"} err="failed to get container status \"a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf\": rpc error: code = NotFound desc = could not find container \"a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf\": container with ID starting with a767477a413305c2f7f96858a066a6eafcf6ce1eb1985ebd3a1ac61015ebd5cf not found: ID does not exist"
Jul 07 12:00:19.343214 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.342422    2565 scope.go:117] "RemoveContainer" containerID="bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1"
Jul 07 12:00:19.343214 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:00:19.342919    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1\": container with ID starting with bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1 not found: ID does not exist" containerID="bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1"
Jul 07 12:00:19.343214 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.342945    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1"} err="failed to get container status \"bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1\": rpc error: code = NotFound desc = could not find container \"bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1\": container with ID starting with bb6124e525b9f33c9679898f894d277d759a78b0f34da39caea1718c89e2a8f1 not found: ID does not exist"
Jul 07 12:00:19.344147 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.343765    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-8255d-predictor-6fdb7dcc4d-fmz78"]
Jul 07 12:00:19.350125 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.350107    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 12:00:19.351782 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.351763    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 12:00:19.889945 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:19.889903    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" path="/var/lib/kubelet/pods/36720b6c-40c2-4611-a629-bf30ea60c21e/volumes"
Jul 07 12:00:20.309710 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:20.309626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerStarted","Data":"acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84"}
Jul 07 12:00:20.309710 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:20.309694    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerStarted","Data":"1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0"}
Jul 07 12:00:20.310184 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:20.310014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:20.310184 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:20.310150    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:20.311418 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:20.311393    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:00:20.331457 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:20.331407    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podStartSLOduration=2.519055558 podStartE2EDuration="6.331392758s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8123372s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:00:20.329514618 +0000 UTC m=+1205.030447479" watchObservedRunningTime="2026-07-07 12:00:20.331392758 +0000 UTC m=+1205.032325617"
Jul 07 12:00:21.314085 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:21.314046    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:00:26.319113 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:26.319085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:00:26.319643 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:26.319618    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:00:36.319626 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:36.319584    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:00:46.319971 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:46.319928    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:00:56.320586 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:00:56.320541    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:01:06.320401 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:06.320357    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:01:16.320091 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:16.320051    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:01:26.319602 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:26.319557    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:01:29.880265 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:29.880223    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 07 12:01:39.884255 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:39.884229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:01:44.391460 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391428    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"]
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391709    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="storage-initializer"
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391721    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="storage-initializer"
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391743    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container"
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391749    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container"
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391757    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kube-rbac-proxy"
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391763    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kube-rbac-proxy"
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391822    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kube-rbac-proxy"
Jul 07 12:01:44.391845 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.391830    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="36720b6c-40c2-4611-a629-bf30ea60c21e" containerName="kserve-container"
Jul 07 12:01:44.394530 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.394513    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.397186 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.397157    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2abe02-kube-rbac-proxy-sar-config\""
Jul 07 12:01:44.397328 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.397191    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-2abe02-predictor-serving-cert\""
Jul 07 12:01:44.397328 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.397237    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-2abe02-dockercfg-pfc27\""
Jul 07 12:01:44.397328 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.397157    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-2abe02\""
Jul 07 12:01:44.404679 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.404650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"]
Jul 07 12:01:44.415210 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.415183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-cabundle-cert\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.415370 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.415252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.415370 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.415286    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a54334f-90bb-4bb9-b79a-17352d8a332a-kserve-provision-location\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.415370 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.415334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.415370 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.415361    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjd5\" (UniqueName: \"kubernetes.io/projected/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-api-access-dzjd5\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.516369 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.516333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-cabundle-cert\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.516582 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.516395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.516582 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.516418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a54334f-90bb-4bb9-b79a-17352d8a332a-kserve-provision-location\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.516582 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.516452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.516582 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.516475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjd5\" (UniqueName: \"kubernetes.io/projected/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-api-access-dzjd5\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.516582 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:01:44.516549    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-serving-cert: secret "isvc-secondary-2abe02-predictor-serving-cert" not found
Jul 07 12:01:44.516861 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:01:44.516634    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls podName:1a54334f-90bb-4bb9-b79a-17352d8a332a nodeName:}" failed. No retries permitted until 2026-07-07 12:01:45.016611252 +0000 UTC m=+1289.717544104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls") pod "isvc-secondary-2abe02-predictor-674dbf54d-6k97n" (UID: "1a54334f-90bb-4bb9-b79a-17352d8a332a") : secret "isvc-secondary-2abe02-predictor-serving-cert" not found
Jul 07 12:01:44.516929 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.516860    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a54334f-90bb-4bb9-b79a-17352d8a332a-kserve-provision-location\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.517145 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.517125    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.517211 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.517128    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-cabundle-cert\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:44.525526 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:44.525499    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjd5\" (UniqueName: \"kubernetes.io/projected/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-api-access-dzjd5\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:45.020131 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:45.020073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:45.022508 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:45.022471    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls\") pod \"isvc-secondary-2abe02-predictor-674dbf54d-6k97n\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") " pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:45.305842 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:45.305747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:01:45.442314 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:45.442250    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"]
Jul 07 12:01:45.445990 ip-10-0-143-227 kubenswrapper[2565]: W0707 12:01:45.445965    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a54334f_90bb_4bb9_b79a_17352d8a332a.slice/crio-e382f02579a69519a9b7104f7fbe89109ff8a5b0b42ffb55b7d04b5dddb4d094 WatchSource:0}: Error finding container e382f02579a69519a9b7104f7fbe89109ff8a5b0b42ffb55b7d04b5dddb4d094: Status 404 returned error can't find the container with id e382f02579a69519a9b7104f7fbe89109ff8a5b0b42ffb55b7d04b5dddb4d094
Jul 07 12:01:45.611912 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:45.611877    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n" event={"ID":"1a54334f-90bb-4bb9-b79a-17352d8a332a","Type":"ContainerStarted","Data":"ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7"}
Jul 07 12:01:45.612080 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:45.611931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n" event={"ID":"1a54334f-90bb-4bb9-b79a-17352d8a332a","Type":"ContainerStarted","Data":"e382f02579a69519a9b7104f7fbe89109ff8a5b0b42ffb55b7d04b5dddb4d094"}
Jul 07 12:01:49.626376 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:49.626344    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_1a54334f-90bb-4bb9-b79a-17352d8a332a/storage-initializer/0.log"
Jul 07 12:01:49.626809 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:49.626384    2565 generic.go:356] "Generic (PLEG): container finished" podID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerID="ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7" exitCode=1
Jul 07 12:01:49.626809 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:49.626433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n" event={"ID":"1a54334f-90bb-4bb9-b79a-17352d8a332a","Type":"ContainerDied","Data":"ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7"}
Jul 07 12:01:50.631686 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:50.631655    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_1a54334f-90bb-4bb9-b79a-17352d8a332a/storage-initializer/0.log"
Jul 07 12:01:50.632098 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:50.631724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n" event={"ID":"1a54334f-90bb-4bb9-b79a-17352d8a332a","Type":"ContainerStarted","Data":"717a53644c347197a3a909fa7fc63344ff02ed708b46519637976dcbbf0682d8"}
Jul 07 12:01:53.644529 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:53.644504    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_1a54334f-90bb-4bb9-b79a-17352d8a332a/storage-initializer/1.log"
Jul 07 12:01:53.644937 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:53.644888    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_1a54334f-90bb-4bb9-b79a-17352d8a332a/storage-initializer/0.log"
Jul 07 12:01:53.644937 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:53.644920    2565 generic.go:356] "Generic (PLEG): container finished" podID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerID="717a53644c347197a3a909fa7fc63344ff02ed708b46519637976dcbbf0682d8" exitCode=1
Jul 07 12:01:53.645013 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:53.644989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n" event={"ID":"1a54334f-90bb-4bb9-b79a-17352d8a332a","Type":"ContainerDied","Data":"717a53644c347197a3a909fa7fc63344ff02ed708b46519637976dcbbf0682d8"}
Jul 07 12:01:53.645059 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:53.645048    2565 scope.go:117] "RemoveContainer" containerID="ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7"
Jul 07 12:01:53.645371 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:53.645350    2565 scope.go:117] "RemoveContainer" containerID="ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7"
Jul 07 12:01:53.655236 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:01:53.655213    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-2abe02-predictor-674dbf54d-6k97n_kserve-ci-e2e-test(1a54334f-90bb-4bb9-b79a-17352d8a332a)\"" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a"
Jul 07 12:01:53.655341 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:01:53.655313    2565 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_kserve-ci-e2e-test_1a54334f-90bb-4bb9-b79a-17352d8a332a_0 in pod sandbox e382f02579a69519a9b7104f7fbe89109ff8a5b0b42ffb55b7d04b5dddb4d094 from index: no such id: 'ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7'" containerID="ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7"
Jul 07 12:01:53.655388 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:53.655353    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_kserve-ci-e2e-test_1a54334f-90bb-4bb9-b79a-17352d8a332a_0 in pod sandbox e382f02579a69519a9b7104f7fbe89109ff8a5b0b42ffb55b7d04b5dddb4d094 from index: no such id: 'ac744bc5efecec5db13bcba04787337593c1c268105cea516f4935017f36f1a7'"
Jul 07 12:01:54.655318 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:01:54.655282    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_1a54334f-90bb-4bb9-b79a-17352d8a332a/storage-initializer/1.log"
Jul 07 12:02:00.420978 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.420941    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"]
Jul 07 12:02:00.479972 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.479940    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"]
Jul 07 12:02:00.480253 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.480225    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container" containerID="cri-o://1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0" gracePeriod=30
Jul 07 12:02:00.480375 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.480265    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kube-rbac-proxy" containerID="cri-o://acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84" gracePeriod=30
Jul 07 12:02:00.598357 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.598335    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_1a54334f-90bb-4bb9-b79a-17352d8a332a/storage-initializer/1.log"
Jul 07 12:02:00.600588 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.600573    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:02:00.643221 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.643191    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a54334f-90bb-4bb9-b79a-17352d8a332a-kserve-provision-location\") pod \"1a54334f-90bb-4bb9-b79a-17352d8a332a\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") "
Jul 07 12:02:00.643356 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.643243    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-cabundle-cert\") pod \"1a54334f-90bb-4bb9-b79a-17352d8a332a\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") "
Jul 07 12:02:00.643356 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.643270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls\") pod \"1a54334f-90bb-4bb9-b79a-17352d8a332a\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") "
Jul 07 12:02:00.643356 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.643288    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzjd5\" (UniqueName: \"kubernetes.io/projected/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-api-access-dzjd5\") pod \"1a54334f-90bb-4bb9-b79a-17352d8a332a\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") "
Jul 07 12:02:00.643356 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.643348    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-rbac-proxy-sar-config\") pod \"1a54334f-90bb-4bb9-b79a-17352d8a332a\" (UID: \"1a54334f-90bb-4bb9-b79a-17352d8a332a\") "
Jul 07 12:02:00.643612 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.643457    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a54334f-90bb-4bb9-b79a-17352d8a332a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a54334f-90bb-4bb9-b79a-17352d8a332a" (UID: "1a54334f-90bb-4bb9-b79a-17352d8a332a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:02:00.643612 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.643573    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a54334f-90bb-4bb9-b79a-17352d8a332a" (UID: "1a54334f-90bb-4bb9-b79a-17352d8a332a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:00.645825 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.644191    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a54334f-90bb-4bb9-b79a-17352d8a332a" (UID: "1a54334f-90bb-4bb9-b79a-17352d8a332a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:00.647521 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.646276    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a54334f-90bb-4bb9-b79a-17352d8a332a" (UID: "1a54334f-90bb-4bb9-b79a-17352d8a332a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:02:00.650167 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.650138    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-api-access-dzjd5" (OuterVolumeSpecName: "kube-api-access-dzjd5") pod "1a54334f-90bb-4bb9-b79a-17352d8a332a" (UID: "1a54334f-90bb-4bb9-b79a-17352d8a332a"). InnerVolumeSpecName "kube-api-access-dzjd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:02:00.678518 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.678453    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-2abe02-predictor-674dbf54d-6k97n_1a54334f-90bb-4bb9-b79a-17352d8a332a/storage-initializer/1.log"
Jul 07 12:02:00.678609 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.678587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n" event={"ID":"1a54334f-90bb-4bb9-b79a-17352d8a332a","Type":"ContainerDied","Data":"e382f02579a69519a9b7104f7fbe89109ff8a5b0b42ffb55b7d04b5dddb4d094"}
Jul 07 12:02:00.678662 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.678621    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"
Jul 07 12:02:00.678704 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.678682    2565 scope.go:117] "RemoveContainer" containerID="717a53644c347197a3a909fa7fc63344ff02ed708b46519637976dcbbf0682d8"
Jul 07 12:02:00.680812 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.680784    2565 generic.go:356] "Generic (PLEG): container finished" podID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerID="acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84" exitCode=2
Jul 07 12:02:00.680905 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.680837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerDied","Data":"acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84"}
Jul 07 12:02:00.713042 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.713014    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"]
Jul 07 12:02:00.716722 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.716701    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-2abe02-predictor-674dbf54d-6k97n"]
Jul 07 12:02:00.744761 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.744737    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a54334f-90bb-4bb9-b79a-17352d8a332a-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:00.744761 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.744757    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzjd5\" (UniqueName: \"kubernetes.io/projected/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-api-access-dzjd5\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:00.744895 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.744767    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:00.744895 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.744777    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a54334f-90bb-4bb9-b79a-17352d8a332a-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:00.744895 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:00.744785    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a54334f-90bb-4bb9-b79a-17352d8a332a-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:01.314545 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:01.314505    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 07 12:02:01.883903 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:01.883873    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a" path="/var/lib/kubelet/pods/1a54334f-90bb-4bb9-b79a-17352d8a332a/volumes"
Jul 07 12:02:05.123874 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.123852    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:02:05.176764 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.176738    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5904b2c6-0388-44b0-bc40-b72a45ffa369-kserve-provision-location\") pod \"5904b2c6-0388-44b0-bc40-b72a45ffa369\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") "
Jul 07 12:02:05.176886 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.176771    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-cabundle-cert\") pod \"5904b2c6-0388-44b0-bc40-b72a45ffa369\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") "
Jul 07 12:02:05.176886 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.176790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4bxq\" (UniqueName: \"kubernetes.io/projected/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-api-access-n4bxq\") pod \"5904b2c6-0388-44b0-bc40-b72a45ffa369\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") "
Jul 07 12:02:05.176886 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.176816    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5904b2c6-0388-44b0-bc40-b72a45ffa369-proxy-tls\") pod \"5904b2c6-0388-44b0-bc40-b72a45ffa369\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") "
Jul 07 12:02:05.176886 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.176852    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-rbac-proxy-sar-config\") pod \"5904b2c6-0388-44b0-bc40-b72a45ffa369\" (UID: \"5904b2c6-0388-44b0-bc40-b72a45ffa369\") "
Jul 07 12:02:05.177155 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.177120    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5904b2c6-0388-44b0-bc40-b72a45ffa369" (UID: "5904b2c6-0388-44b0-bc40-b72a45ffa369"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:05.177276 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.177165    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5904b2c6-0388-44b0-bc40-b72a45ffa369-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5904b2c6-0388-44b0-bc40-b72a45ffa369" (UID: "5904b2c6-0388-44b0-bc40-b72a45ffa369"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:02:05.177276 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.177254    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5904b2c6-0388-44b0-bc40-b72a45ffa369" (UID: "5904b2c6-0388-44b0-bc40-b72a45ffa369"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:05.178785 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.178760    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-api-access-n4bxq" (OuterVolumeSpecName: "kube-api-access-n4bxq") pod "5904b2c6-0388-44b0-bc40-b72a45ffa369" (UID: "5904b2c6-0388-44b0-bc40-b72a45ffa369"). InnerVolumeSpecName "kube-api-access-n4bxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:02:05.178871 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.178802    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5904b2c6-0388-44b0-bc40-b72a45ffa369-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5904b2c6-0388-44b0-bc40-b72a45ffa369" (UID: "5904b2c6-0388-44b0-bc40-b72a45ffa369"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:02:05.278327 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.278259    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:05.278327 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.278284    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5904b2c6-0388-44b0-bc40-b72a45ffa369-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:05.278327 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.278294    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5904b2c6-0388-44b0-bc40-b72a45ffa369-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:05.278327 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.278304    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bxq\" (UniqueName: \"kubernetes.io/projected/5904b2c6-0388-44b0-bc40-b72a45ffa369-kube-api-access-n4bxq\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:05.278327 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.278315    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5904b2c6-0388-44b0-bc40-b72a45ffa369-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:05.575112 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575040    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"]
Jul 07 12:02:05.575303 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575291    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerName="storage-initializer"
Jul 07 12:02:05.575345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575304    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerName="storage-initializer"
Jul 07 12:02:05.575345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575312    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerName="storage-initializer"
Jul 07 12:02:05.575345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575318    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerName="storage-initializer"
Jul 07 12:02:05.575345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575329    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container"
Jul 07 12:02:05.575345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575334    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container"
Jul 07 12:02:05.575345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575341    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kube-rbac-proxy"
Jul 07 12:02:05.575345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575347    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kube-rbac-proxy"
Jul 07 12:02:05.575571 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575355    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="storage-initializer"
Jul 07 12:02:05.575571 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575360    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="storage-initializer"
Jul 07 12:02:05.575571 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575397    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kserve-container"
Jul 07 12:02:05.575571 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575405    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerName="storage-initializer"
Jul 07 12:02:05.575571 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575412    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a54334f-90bb-4bb9-b79a-17352d8a332a" containerName="storage-initializer"
Jul 07 12:02:05.575571 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.575419    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerName="kube-rbac-proxy"
Jul 07 12:02:05.578626 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.578603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.581330 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.581266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e57645-predictor-serving-cert\""
Jul 07 12:02:05.581450 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.581329    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e57645-kube-rbac-proxy-sar-config\""
Jul 07 12:02:05.581450 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.581342    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e57645\""
Jul 07 12:02:05.581450 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.581352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e57645-dockercfg-n9thf\""
Jul 07 12:02:05.588547 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.588525    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"]
Jul 07 12:02:05.680915 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.680878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-cabundle-cert\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.681075 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.680924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.681075 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.680998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wpw\" (UniqueName: \"kubernetes.io/projected/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-api-access-w7wpw\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.681075 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.681026    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kserve-provision-location\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.681075 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.681056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.700841 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.700807    2565 generic.go:356] "Generic (PLEG): container finished" podID="5904b2c6-0388-44b0-bc40-b72a45ffa369" containerID="1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0" exitCode=0
Jul 07 12:02:05.701017 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.700869    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"
Jul 07 12:02:05.701017 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.700892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerDied","Data":"1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0"}
Jul 07 12:02:05.701017 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.700966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8" event={"ID":"5904b2c6-0388-44b0-bc40-b72a45ffa369","Type":"ContainerDied","Data":"43b0bbe96b1a1428f6a3609166188a0b6e8c48a2e19e524624277204d52efb02"}
Jul 07 12:02:05.701152 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.701013    2565 scope.go:117] "RemoveContainer" containerID="acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84"
Jul 07 12:02:05.709824 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.709726    2565 scope.go:117] "RemoveContainer" containerID="1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0"
Jul 07 12:02:05.717231 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.717216    2565 scope.go:117] "RemoveContainer" containerID="7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b"
Jul 07 12:02:05.722758 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.722735    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"]
Jul 07 12:02:05.725180 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.725150    2565 scope.go:117] "RemoveContainer" containerID="acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84"
Jul 07 12:02:05.725530 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:05.725469    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84\": container with ID starting with acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84 not found: ID does not exist" containerID="acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84"
Jul 07 12:02:05.725620 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.725522    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84"} err="failed to get container status \"acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84\": rpc error: code = NotFound desc = could not find container \"acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84\": container with ID starting with acb95255156af2e475ef2156eaea61ac2fcedea8a9366611047e95a93454fc84 not found: ID does not exist"
Jul 07 12:02:05.725620 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.725544    2565 scope.go:117] "RemoveContainer" containerID="1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0"
Jul 07 12:02:05.725831 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:05.725810    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0\": container with ID starting with 1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0 not found: ID does not exist" containerID="1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0"
Jul 07 12:02:05.725885 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.725840    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0"} err="failed to get container status \"1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0\": rpc error: code = NotFound desc = could not find container \"1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0\": container with ID starting with 1a68dbc03b0a65aec406909939cb17b7ecab347b39a5cd1f12b96008098830b0 not found: ID does not exist"
Jul 07 12:02:05.725932 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.725891    2565 scope.go:117] "RemoveContainer" containerID="7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b"
Jul 07 12:02:05.726154 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:05.726137    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b\": container with ID starting with 7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b not found: ID does not exist" containerID="7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b"
Jul 07 12:02:05.726202 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.726158    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b"} err="failed to get container status \"7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b\": rpc error: code = NotFound desc = could not find container \"7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b\": container with ID starting with 7e3d742b1a5d37b7929bc7f53e2e70f6426d24ad3c2f464a291dee9e7687539b not found: ID does not exist"
Jul 07 12:02:05.726238 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.726216    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-2abe02-predictor-66db8475c5-29lx8"]
Jul 07 12:02:05.781676 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.781637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wpw\" (UniqueName: \"kubernetes.io/projected/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-api-access-w7wpw\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.781676 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.781674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kserve-provision-location\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.781900 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.781707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.781900 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.781766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-cabundle-cert\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.781900 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.781794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.782047 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:05.781903    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-serving-cert: secret "isvc-init-fail-e57645-predictor-serving-cert" not found
Jul 07 12:02:05.782047 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:05.781973    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls podName:8e35eb1c-8f34-4ea2-b5b0-29175a1040c6 nodeName:}" failed. No retries permitted until 2026-07-07 12:02:06.281951066 +0000 UTC m=+1310.982883904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls") pod "isvc-init-fail-e57645-predictor-7785c44686-kssvg" (UID: "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6") : secret "isvc-init-fail-e57645-predictor-serving-cert" not found
Jul 07 12:02:05.782176 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.782074    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kserve-provision-location\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.782447 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.782425    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-cabundle-cert\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.782548 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.782470    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.793214 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.793189    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wpw\" (UniqueName: \"kubernetes.io/projected/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-api-access-w7wpw\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:05.884743 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:05.884712    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5904b2c6-0388-44b0-bc40-b72a45ffa369" path="/var/lib/kubelet/pods/5904b2c6-0388-44b0-bc40-b72a45ffa369/volumes"
Jul 07 12:02:06.285542 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:06.285431    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:06.287731 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:06.287708    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls\") pod \"isvc-init-fail-e57645-predictor-7785c44686-kssvg\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") " pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:06.490774 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:06.490732    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:06.619900 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:06.619878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"]
Jul 07 12:02:06.622460 ip-10-0-143-227 kubenswrapper[2565]: W0707 12:02:06.622428    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e35eb1c_8f34_4ea2_b5b0_29175a1040c6.slice/crio-f228e3a7d267da1298857a8df499b4faf5dadfda91d57e8f2f47f9608ce8457b WatchSource:0}: Error finding container f228e3a7d267da1298857a8df499b4faf5dadfda91d57e8f2f47f9608ce8457b: Status 404 returned error can't find the container with id f228e3a7d267da1298857a8df499b4faf5dadfda91d57e8f2f47f9608ce8457b
Jul 07 12:02:06.707386 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:06.707355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg" event={"ID":"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6","Type":"ContainerStarted","Data":"c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec"}
Jul 07 12:02:06.707530 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:06.707404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg" event={"ID":"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6","Type":"ContainerStarted","Data":"f228e3a7d267da1298857a8df499b4faf5dadfda91d57e8f2f47f9608ce8457b"}
Jul 07 12:02:09.720797 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:09.720769    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e57645-predictor-7785c44686-kssvg_8e35eb1c-8f34-4ea2-b5b0-29175a1040c6/storage-initializer/0.log"
Jul 07 12:02:09.721174 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:09.720808    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerID="c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec" exitCode=1
Jul 07 12:02:09.721174 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:09.720854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg" event={"ID":"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6","Type":"ContainerDied","Data":"c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec"}
Jul 07 12:02:10.594712 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:10.594678    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"]
Jul 07 12:02:10.725534 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:10.725507    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e57645-predictor-7785c44686-kssvg_8e35eb1c-8f34-4ea2-b5b0-29175a1040c6/storage-initializer/0.log"
Jul 07 12:02:10.725956 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:10.725615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg" event={"ID":"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6","Type":"ContainerStarted","Data":"798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474"}
Jul 07 12:02:10.725956 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:10.725697    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerName="storage-initializer" containerID="cri-o://798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474" gracePeriod=30
Jul 07 12:02:14.064136 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.064113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e57645-predictor-7785c44686-kssvg_8e35eb1c-8f34-4ea2-b5b0-29175a1040c6/storage-initializer/1.log"
Jul 07 12:02:14.064566 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.064549    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e57645-predictor-7785c44686-kssvg_8e35eb1c-8f34-4ea2-b5b0-29175a1040c6/storage-initializer/0.log"
Jul 07 12:02:14.066407 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.066391    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:14.144755 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.144663    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-cabundle-cert\") pod \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") "
Jul 07 12:02:14.144755 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.144702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7wpw\" (UniqueName: \"kubernetes.io/projected/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-api-access-w7wpw\") pod \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") "
Jul 07 12:02:14.144755 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.144727    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls\") pod \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") "
Jul 07 12:02:14.145016 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.144809    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kserve-provision-location\") pod \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") "
Jul 07 12:02:14.145016 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.144844    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-rbac-proxy-sar-config\") pod \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\" (UID: \"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6\") "
Jul 07 12:02:14.145162 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.145106    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" (UID: "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:02:14.145262 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.145118    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" (UID: "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:14.145309 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.145261    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" (UID: "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:02:14.146798 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.146773    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" (UID: "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:02:14.146891 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.146869    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-api-access-w7wpw" (OuterVolumeSpecName: "kube-api-access-w7wpw") pod "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" (UID: "8e35eb1c-8f34-4ea2-b5b0-29175a1040c6"). InnerVolumeSpecName "kube-api-access-w7wpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:02:14.245594 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.245562    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:14.245594 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.245588    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:14.245594 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.245597    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:14.245829 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.245606    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7wpw\" (UniqueName: \"kubernetes.io/projected/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-kube-api-access-w7wpw\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:14.245829 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.245615    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:02:14.740892 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.740866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e57645-predictor-7785c44686-kssvg_8e35eb1c-8f34-4ea2-b5b0-29175a1040c6/storage-initializer/1.log"
Jul 07 12:02:14.741283 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.741266    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e57645-predictor-7785c44686-kssvg_8e35eb1c-8f34-4ea2-b5b0-29175a1040c6/storage-initializer/0.log"
Jul 07 12:02:14.741392 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.741303    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerID="798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474" exitCode=1
Jul 07 12:02:14.741392 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.741358    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"
Jul 07 12:02:14.741509 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.741351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg" event={"ID":"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6","Type":"ContainerDied","Data":"798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474"}
Jul 07 12:02:14.741562 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.741511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg" event={"ID":"8e35eb1c-8f34-4ea2-b5b0-29175a1040c6","Type":"ContainerDied","Data":"f228e3a7d267da1298857a8df499b4faf5dadfda91d57e8f2f47f9608ce8457b"}
Jul 07 12:02:14.741609 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.741559    2565 scope.go:117] "RemoveContainer" containerID="798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474"
Jul 07 12:02:14.749598 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.749581    2565 scope.go:117] "RemoveContainer" containerID="c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec"
Jul 07 12:02:14.756499 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.756456    2565 scope.go:117] "RemoveContainer" containerID="798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474"
Jul 07 12:02:14.756731 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:14.756712    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474\": container with ID starting with 798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474 not found: ID does not exist" containerID="798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474"
Jul 07 12:02:14.756800 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.756743    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474"} err="failed to get container status \"798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474\": rpc error: code = NotFound desc = could not find container \"798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474\": container with ID starting with 798ab024fccc919047043ed6406bf206395081ed326c3cdea20f94d4ee564474 not found: ID does not exist"
Jul 07 12:02:14.756800 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.756767    2565 scope.go:117] "RemoveContainer" containerID="c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec"
Jul 07 12:02:14.757026 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:14.757010    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec\": container with ID starting with c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec not found: ID does not exist" containerID="c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec"
Jul 07 12:02:14.757063 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.757032    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec"} err="failed to get container status \"c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec\": rpc error: code = NotFound desc = could not find container \"c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec\": container with ID starting with c762884bb698079482f4e3948d242e002f39099859de9669725b5ea655ba78ec not found: ID does not exist"
Jul 07 12:02:14.775599 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.775574    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"]
Jul 07 12:02:14.778900 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:14.778879    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e57645-predictor-7785c44686-kssvg"]
Jul 07 12:02:15.770699 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.770662    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"]
Jul 07 12:02:15.771068 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.770911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerName="storage-initializer"
Jul 07 12:02:15.771068 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.770922    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerName="storage-initializer"
Jul 07 12:02:15.771068 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.770974    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerName="storage-initializer"
Jul 07 12:02:15.771068 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.770983    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerName="storage-initializer"
Jul 07 12:02:15.771068 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.771029    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerName="storage-initializer"
Jul 07 12:02:15.771068 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.771035    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" containerName="storage-initializer"
Jul 07 12:02:15.774934 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.774913    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.777373 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.777350    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 12:02:15.777647 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.777632    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-acf07-predictor-serving-cert\""
Jul 07 12:02:15.777741 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.777713    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 12:02:15.777809 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.777747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lhlhn\""
Jul 07 12:02:15.777809 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.777772    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 12:02:15.777922 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.777904    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-acf07-kube-rbac-proxy-sar-config\""
Jul 07 12:02:15.778702 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.778666    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 12:02:15.784875 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.784807    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"]
Jul 07 12:02:15.858313 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.858286    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-cabundle-cert\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.858406 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.858339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.858406 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.858366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdvcs\" (UniqueName: \"kubernetes.io/projected/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-api-access-vdvcs\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.858406 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.858392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kserve-provision-location\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.858537 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.858427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.884741 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.884713    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e35eb1c-8f34-4ea2-b5b0-29175a1040c6" path="/var/lib/kubelet/pods/8e35eb1c-8f34-4ea2-b5b0-29175a1040c6/volumes"
Jul 07 12:02:15.959311 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.959270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kserve-provision-location\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.959510 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.959338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.959510 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.959422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-cabundle-cert\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.959641 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.959528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.959641 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.959562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdvcs\" (UniqueName: \"kubernetes.io/projected/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-api-access-vdvcs\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.959744 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:15.959732    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-acf07-predictor-serving-cert: secret "raw-sklearn-acf07-predictor-serving-cert" not found
Jul 07 12:02:15.959793 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.959752    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kserve-provision-location\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.959793 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:02:15.959785    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls podName:17fa0a01-2986-4bf1-b72c-54fd07d41a7f nodeName:}" failed. No retries permitted until 2026-07-07 12:02:16.459765093 +0000 UTC m=+1321.160697935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls") pod "raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" (UID: "17fa0a01-2986-4bf1-b72c-54fd07d41a7f") : secret "raw-sklearn-acf07-predictor-serving-cert" not found
Jul 07 12:02:15.960085 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.960067    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.960296 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.960277    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-cabundle-cert\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:15.969922 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:15.969899    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdvcs\" (UniqueName: \"kubernetes.io/projected/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-api-access-vdvcs\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:16.464041 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:16.463995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:16.466313 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:16.466293    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls\") pod \"raw-sklearn-acf07-predictor-7bcfb88794-x5xcw\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") " pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:16.687253 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:16.687215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:16.827313 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:16.827291    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"]
Jul 07 12:02:16.829531 ip-10-0-143-227 kubenswrapper[2565]: W0707 12:02:16.829505    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fa0a01_2986_4bf1_b72c_54fd07d41a7f.slice/crio-4b492a51604ba585496ccc66c4a9cb4eba785c219db6daff7addad635fd677f4 WatchSource:0}: Error finding container 4b492a51604ba585496ccc66c4a9cb4eba785c219db6daff7addad635fd677f4: Status 404 returned error can't find the container with id 4b492a51604ba585496ccc66c4a9cb4eba785c219db6daff7addad635fd677f4
Jul 07 12:02:17.756415 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:17.756379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerStarted","Data":"ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec"}
Jul 07 12:02:17.756595 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:17.756436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerStarted","Data":"4b492a51604ba585496ccc66c4a9cb4eba785c219db6daff7addad635fd677f4"}
Jul 07 12:02:20.768289 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:20.768250    2565 generic.go:356] "Generic (PLEG): container finished" podID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerID="ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec" exitCode=0
Jul 07 12:02:20.768786 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:20.768330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerDied","Data":"ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec"}
Jul 07 12:02:21.773943 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:21.773904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerStarted","Data":"18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0"}
Jul 07 12:02:21.774291 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:21.773993    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerStarted","Data":"8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c"}
Jul 07 12:02:21.774291 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:21.774184    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:21.795149 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:21.795102    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podStartSLOduration=3.152542966 podStartE2EDuration="6.795084065s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.642541099s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:02:21.794000176 +0000 UTC m=+1326.494933036" watchObservedRunningTime="2026-07-07 12:02:21.795084065 +0000 UTC m=+1326.496016925"
Jul 07 12:02:22.777776 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:22.777740    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:22.778986 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:22.778961    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:23.781387 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:23.781346    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:28.786474 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:28.786445    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:02:28.786945 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:28.786921    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:38.787329 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:38.787246    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:48.787600 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:48.787561    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:02:58.787017 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:02:58.786977    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:03:08.787227 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:08.787191    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:03:18.787426 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:18.787392    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:03:28.787158 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:28.787116    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:03:38.788121 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:38.788093    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:03:45.900938 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:45.900901    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"]
Jul 07 12:03:45.901427 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:45.901213    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" containerID="cri-o://8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c" gracePeriod=30
Jul 07 12:03:45.901427 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:45.901226    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kube-rbac-proxy" containerID="cri-o://18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0" gracePeriod=30
Jul 07 12:03:46.061170 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:46.061137    2565 generic.go:356] "Generic (PLEG): container finished" podID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerID="18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0" exitCode=2
Jul 07 12:03:46.061310 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:46.061212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerDied","Data":"18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0"}
Jul 07 12:03:48.781620 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:48.781583    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 07 12:03:48.787497 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:48.787449    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 07 12:03:50.749358 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.749332    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:03:50.881393 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881362    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls\") pod \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") "
Jul 07 12:03:50.881608 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881404    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-cabundle-cert\") pod \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") "
Jul 07 12:03:50.881608 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881425    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-rbac-proxy-sar-config\") pod \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") "
Jul 07 12:03:50.881608 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881469    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kserve-provision-location\") pod \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") "
Jul 07 12:03:50.881608 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdvcs\" (UniqueName: \"kubernetes.io/projected/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-api-access-vdvcs\") pod \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\" (UID: \"17fa0a01-2986-4bf1-b72c-54fd07d41a7f\") "
Jul 07 12:03:50.881889 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881860    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17fa0a01-2986-4bf1-b72c-54fd07d41a7f" (UID: "17fa0a01-2986-4bf1-b72c-54fd07d41a7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:03:50.882005 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881918    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17fa0a01-2986-4bf1-b72c-54fd07d41a7f" (UID: "17fa0a01-2986-4bf1-b72c-54fd07d41a7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:03:50.882005 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.881967    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17fa0a01-2986-4bf1-b72c-54fd07d41a7f" (UID: "17fa0a01-2986-4bf1-b72c-54fd07d41a7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:03:50.883655 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.883630    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17fa0a01-2986-4bf1-b72c-54fd07d41a7f" (UID: "17fa0a01-2986-4bf1-b72c-54fd07d41a7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:03:50.883780 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.883763    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-api-access-vdvcs" (OuterVolumeSpecName: "kube-api-access-vdvcs") pod "17fa0a01-2986-4bf1-b72c-54fd07d41a7f" (UID: "17fa0a01-2986-4bf1-b72c-54fd07d41a7f"). InnerVolumeSpecName "kube-api-access-vdvcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:03:50.982904 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.982859    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:03:50.982904 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.982896    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdvcs\" (UniqueName: \"kubernetes.io/projected/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-api-access-vdvcs\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:03:50.982904 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.982911    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:03:50.983140 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.982924    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:03:50.983140 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:50.982939    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17fa0a01-2986-4bf1-b72c-54fd07d41a7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:03:51.024927 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.024892    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"]
Jul 07 12:03:51.025190 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025176    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kube-rbac-proxy"
Jul 07 12:03:51.025233 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025194    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kube-rbac-proxy"
Jul 07 12:03:51.025233 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025205    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container"
Jul 07 12:03:51.025233 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025211    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container"
Jul 07 12:03:51.025233 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025217    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="storage-initializer"
Jul 07 12:03:51.025233 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025224    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="storage-initializer"
Jul 07 12:03:51.025379 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025278    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kserve-container"
Jul 07 12:03:51.025379 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.025286    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerName="kube-rbac-proxy"
Jul 07 12:03:51.028344 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.028324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.031038 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.031013    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-7e4c0-predictor-serving-cert\""
Jul 07 12:03:51.031038 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.031032    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-7e4c0-kube-rbac-proxy-sar-config\""
Jul 07 12:03:51.041166 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.041136    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"]
Jul 07 12:03:51.083713 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.083663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97258ce3-7359-4398-a5f4-6155a6fdefdd-kserve-provision-location\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.083906 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.083741    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.083906 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.083846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-cabundle-cert\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.084014 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.083904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97258ce3-7359-4398-a5f4-6155a6fdefdd-proxy-tls\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.084014 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.083928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjsc\" (UniqueName: \"kubernetes.io/projected/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-api-access-dzjsc\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.085320 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.085296    2565 generic.go:356] "Generic (PLEG): container finished" podID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" containerID="8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c" exitCode=0
Jul 07 12:03:51.085428 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.085362    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"
Jul 07 12:03:51.085428 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.085381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerDied","Data":"8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c"}
Jul 07 12:03:51.085534 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.085507    2565 scope.go:117] "RemoveContainer" containerID="18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0"
Jul 07 12:03:51.085626 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.085506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw" event={"ID":"17fa0a01-2986-4bf1-b72c-54fd07d41a7f","Type":"ContainerDied","Data":"4b492a51604ba585496ccc66c4a9cb4eba785c219db6daff7addad635fd677f4"}
Jul 07 12:03:51.093761 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.093649    2565 scope.go:117] "RemoveContainer" containerID="8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c"
Jul 07 12:03:51.101212 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.101195    2565 scope.go:117] "RemoveContainer" containerID="ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec"
Jul 07 12:03:51.108145 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.108130    2565 scope.go:117] "RemoveContainer" containerID="18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0"
Jul 07 12:03:51.108418 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:03:51.108397    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0\": container with ID starting with 18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0 not found: ID does not exist" containerID="18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0"
Jul 07 12:03:51.108465 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.108424    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0"} err="failed to get container status \"18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0\": rpc error: code = NotFound desc = could not find container \"18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0\": container with ID starting with 18d49a193102370f6129538ae86775cbc5d404628f9a51159e6f806bf6a817d0 not found: ID does not exist"
Jul 07 12:03:51.108465 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.108440    2565 scope.go:117] "RemoveContainer" containerID="8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c"
Jul 07 12:03:51.108733 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:03:51.108714    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c\": container with ID starting with 8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c not found: ID does not exist" containerID="8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c"
Jul 07 12:03:51.108782 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.108740    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c"} err="failed to get container status \"8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c\": rpc error: code = NotFound desc = could not find container \"8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c\": container with ID starting with 8282a6caa9de432c24076dedbe840eb5f88ca257df9ba3b7acea026b9ab7206c not found: ID does not exist"
Jul 07 12:03:51.108782 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.108756    2565 scope.go:117] "RemoveContainer" containerID="ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec"
Jul 07 12:03:51.108996 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:03:51.108980    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec\": container with ID starting with ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec not found: ID does not exist" containerID="ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec"
Jul 07 12:03:51.109045 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.108999    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec"} err="failed to get container status \"ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec\": rpc error: code = NotFound desc = could not find container \"ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec\": container with ID starting with ed3f396340dd0447e4e7210260750302ad9bc9957c21dc0754712a55b642ecec not found: ID does not exist"
Jul 07 12:03:51.125367 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.125338    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"]
Jul 07 12:03:51.129924 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.129901    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-acf07-predictor-7bcfb88794-x5xcw"]
Jul 07 12:03:51.184963 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.184868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-cabundle-cert\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.184963 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.184923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97258ce3-7359-4398-a5f4-6155a6fdefdd-proxy-tls\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.184963 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.184951    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjsc\" (UniqueName: \"kubernetes.io/projected/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-api-access-dzjsc\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.185228 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.184980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97258ce3-7359-4398-a5f4-6155a6fdefdd-kserve-provision-location\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.185228 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.185006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.185505 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.185457    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97258ce3-7359-4398-a5f4-6155a6fdefdd-kserve-provision-location\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.185775 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.185749    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-cabundle-cert\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.185898 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.185749    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.187541 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.187523    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97258ce3-7359-4398-a5f4-6155a6fdefdd-proxy-tls\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.194213 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.194189    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjsc\" (UniqueName: \"kubernetes.io/projected/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-api-access-dzjsc\") pod \"raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.339307 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.339266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:51.466327 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.466304    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"]
Jul 07 12:03:51.468352 ip-10-0-143-227 kubenswrapper[2565]: W0707 12:03:51.468331    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97258ce3_7359_4398_a5f4_6155a6fdefdd.slice/crio-293c282bdb72f8bea418f447b9c179cb29cf055b23b6e6c242480564dd5f4e3c WatchSource:0}: Error finding container 293c282bdb72f8bea418f447b9c179cb29cf055b23b6e6c242480564dd5f4e3c: Status 404 returned error can't find the container with id 293c282bdb72f8bea418f447b9c179cb29cf055b23b6e6c242480564dd5f4e3c
Jul 07 12:03:51.884848 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:51.884812    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fa0a01-2986-4bf1-b72c-54fd07d41a7f" path="/var/lib/kubelet/pods/17fa0a01-2986-4bf1-b72c-54fd07d41a7f/volumes"
Jul 07 12:03:52.091256 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:52.091217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerStarted","Data":"22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e"}
Jul 07 12:03:52.091413 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:52.091292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerStarted","Data":"293c282bdb72f8bea418f447b9c179cb29cf055b23b6e6c242480564dd5f4e3c"}
Jul 07 12:03:56.106002 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:56.105966    2565 generic.go:356] "Generic (PLEG): container finished" podID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerID="22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e" exitCode=0
Jul 07 12:03:56.106526 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:56.106051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerDied","Data":"22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e"}
Jul 07 12:03:57.112144 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:57.112106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerStarted","Data":"473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73"}
Jul 07 12:03:57.112572 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:57.112192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerStarted","Data":"2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6"}
Jul 07 12:03:57.112572 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:57.112376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:57.133841 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:57.133797    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podStartSLOduration=2.243824373 podStartE2EDuration="6.133784143s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.88995977s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:03:57.130425349 +0000 UTC m=+1421.831358209" watchObservedRunningTime="2026-07-07 12:03:57.133784143 +0000 UTC m=+1421.834717001"
Jul 07 12:03:58.115565 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:58.115534    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:03:58.116943 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:58.116917    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:03:59.119248 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:03:59.119203    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:04:04.123855 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:04:04.123826    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:04:04.124519 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:04:04.124470    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:04:14.125113 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:04:14.125071    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:04:24.125207 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:04:24.125164    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:04:34.125288 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:04:34.125252    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:04:44.124466 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:04:44.124423    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:04:54.124603 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:04:54.124568    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:05:04.125076 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:04.125034    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 07 12:05:09.884062 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:09.884033    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:05:11.076608 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:11.076572    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"]
Jul 07 12:05:11.372960 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:11.372922    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container" containerID="cri-o://2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6" gracePeriod=30
Jul 07 12:05:11.372960 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:11.372935    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kube-rbac-proxy" containerID="cri-o://473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73" gracePeriod=30
Jul 07 12:05:12.382412 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:12.382381    2565 generic.go:356] "Generic (PLEG): container finished" podID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerID="473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73" exitCode=2
Jul 07 12:05:12.382796 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:12.382436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerDied","Data":"473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73"}
Jul 07 12:05:14.119983 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:14.119941    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 07 12:05:16.119471 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.119448    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:05:16.208616 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.208588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97258ce3-7359-4398-a5f4-6155a6fdefdd-kserve-provision-location\") pod \"97258ce3-7359-4398-a5f4-6155a6fdefdd\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") "
Jul 07 12:05:16.208756 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.208632    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-rbac-proxy-sar-config\") pod \"97258ce3-7359-4398-a5f4-6155a6fdefdd\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") "
Jul 07 12:05:16.208756 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.208651    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-cabundle-cert\") pod \"97258ce3-7359-4398-a5f4-6155a6fdefdd\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") "
Jul 07 12:05:16.208756 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.208690    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97258ce3-7359-4398-a5f4-6155a6fdefdd-proxy-tls\") pod \"97258ce3-7359-4398-a5f4-6155a6fdefdd\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") "
Jul 07 12:05:16.208756 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.208714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzjsc\" (UniqueName: \"kubernetes.io/projected/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-api-access-dzjsc\") pod \"97258ce3-7359-4398-a5f4-6155a6fdefdd\" (UID: \"97258ce3-7359-4398-a5f4-6155a6fdefdd\") "
Jul 07 12:05:16.209028 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.208963    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97258ce3-7359-4398-a5f4-6155a6fdefdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97258ce3-7359-4398-a5f4-6155a6fdefdd" (UID: "97258ce3-7359-4398-a5f4-6155a6fdefdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 12:05:16.209028 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.208990    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97258ce3-7359-4398-a5f4-6155a6fdefdd" (UID: "97258ce3-7359-4398-a5f4-6155a6fdefdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:05:16.209028 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.209021    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97258ce3-7359-4398-a5f4-6155a6fdefdd" (UID: "97258ce3-7359-4398-a5f4-6155a6fdefdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 12:05:16.210809 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.210759    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-api-access-dzjsc" (OuterVolumeSpecName: "kube-api-access-dzjsc") pod "97258ce3-7359-4398-a5f4-6155a6fdefdd" (UID: "97258ce3-7359-4398-a5f4-6155a6fdefdd"). InnerVolumeSpecName "kube-api-access-dzjsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 12:05:16.210809 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.210802    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97258ce3-7359-4398-a5f4-6155a6fdefdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97258ce3-7359-4398-a5f4-6155a6fdefdd" (UID: "97258ce3-7359-4398-a5f4-6155a6fdefdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 12:05:16.309304 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.309260    2565 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97258ce3-7359-4398-a5f4-6155a6fdefdd-proxy-tls\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:05:16.309304 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.309297    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzjsc\" (UniqueName: \"kubernetes.io/projected/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-api-access-dzjsc\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:05:16.309304 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.309311    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97258ce3-7359-4398-a5f4-6155a6fdefdd-kserve-provision-location\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:05:16.309304 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.309321    2565 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:05:16.309575 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.309330    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97258ce3-7359-4398-a5f4-6155a6fdefdd-cabundle-cert\") on node \"ip-10-0-143-227.ec2.internal\" DevicePath \"\""
Jul 07 12:05:16.398669 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.398637    2565 generic.go:356] "Generic (PLEG): container finished" podID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerID="2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6" exitCode=0
Jul 07 12:05:16.398812 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.398669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerDied","Data":"2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6"}
Jul 07 12:05:16.398812 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.398700    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"
Jul 07 12:05:16.398812 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.398733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5" event={"ID":"97258ce3-7359-4398-a5f4-6155a6fdefdd","Type":"ContainerDied","Data":"293c282bdb72f8bea418f447b9c179cb29cf055b23b6e6c242480564dd5f4e3c"}
Jul 07 12:05:16.398812 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.398760    2565 scope.go:117] "RemoveContainer" containerID="473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73"
Jul 07 12:05:16.407877 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.407856    2565 scope.go:117] "RemoveContainer" containerID="2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6"
Jul 07 12:05:16.415778 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.415761    2565 scope.go:117] "RemoveContainer" containerID="22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e"
Jul 07 12:05:16.422231 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.422213    2565 scope.go:117] "RemoveContainer" containerID="473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73"
Jul 07 12:05:16.422472 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:05:16.422454    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73\": container with ID starting with 473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73 not found: ID does not exist" containerID="473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73"
Jul 07 12:05:16.422563 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.422498    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73"} err="failed to get container status \"473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73\": rpc error: code = NotFound desc = could not find container \"473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73\": container with ID starting with 473bc688d667d1e0629f14d375300b2e357d19a6ae0d73f396cc4e899ee3fa73 not found: ID does not exist"
Jul 07 12:05:16.422563 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.422524    2565 scope.go:117] "RemoveContainer" containerID="2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6"
Jul 07 12:05:16.422760 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:05:16.422743    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6\": container with ID starting with 2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6 not found: ID does not exist" containerID="2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6"
Jul 07 12:05:16.422802 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.422766    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6"} err="failed to get container status \"2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6\": rpc error: code = NotFound desc = could not find container \"2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6\": container with ID starting with 2799640a98df0151306a4734444da08fe4a11df2e4b6fb0f17b6b591044eeda6 not found: ID does not exist"
Jul 07 12:05:16.422802 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.422781    2565 scope.go:117] "RemoveContainer" containerID="22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e"
Jul 07 12:05:16.422997 ip-10-0-143-227 kubenswrapper[2565]: E0707 12:05:16.422982    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e\": container with ID starting with 22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e not found: ID does not exist" containerID="22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e"
Jul 07 12:05:16.423036 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.423000    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e"} err="failed to get container status \"22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e\": rpc error: code = NotFound desc = could not find container \"22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e\": container with ID starting with 22e25279f65312cf495c14bd37dcc77e73d5ebdf4ffa8e605eab0aaf40d0aa5e not found: ID does not exist"
Jul 07 12:05:16.426802 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.426781    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"]
Jul 07 12:05:16.430631 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:16.430612    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-7e4c0-predictor-864f7f9ddb-qvvc5"]
Jul 07 12:05:17.884545 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:17.884515    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" path="/var/lib/kubelet/pods/97258ce3-7359-4398-a5f4-6155a6fdefdd/volumes"
Jul 07 12:05:19.371812 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:19.371788    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 12:05:19.376034 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:19.376010    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 12:05:42.898225 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898142    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n47qj/must-gather-fjfcc"]
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898395    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="storage-initializer"
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898405    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="storage-initializer"
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898415    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kube-rbac-proxy"
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898420    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kube-rbac-proxy"
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898434    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container"
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898440    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container"
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898491    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kserve-container"
Jul 07 12:05:42.898691 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.898501    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97258ce3-7359-4398-a5f4-6155a6fdefdd" containerName="kube-rbac-proxy"
Jul 07 12:05:42.903018 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.903001    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:42.905562 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.905535    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n47qj\"/\"openshift-service-ca.crt\""
Jul 07 12:05:42.905678 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.905648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n47qj\"/\"kube-root-ca.crt\""
Jul 07 12:05:42.910854 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:42.910831    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n47qj/must-gather-fjfcc"]
Jul 07 12:05:43.009146 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.009111    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b024b5-d23c-4315-ad97-bed665cac395-must-gather-output\") pod \"must-gather-fjfcc\" (UID: \"92b024b5-d23c-4315-ad97-bed665cac395\") " pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:43.009345 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.009154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8fv9\" (UniqueName: \"kubernetes.io/projected/92b024b5-d23c-4315-ad97-bed665cac395-kube-api-access-q8fv9\") pod \"must-gather-fjfcc\" (UID: \"92b024b5-d23c-4315-ad97-bed665cac395\") " pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:43.110462 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.110432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b024b5-d23c-4315-ad97-bed665cac395-must-gather-output\") pod \"must-gather-fjfcc\" (UID: \"92b024b5-d23c-4315-ad97-bed665cac395\") " pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:43.110574 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.110472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8fv9\" (UniqueName: \"kubernetes.io/projected/92b024b5-d23c-4315-ad97-bed665cac395-kube-api-access-q8fv9\") pod \"must-gather-fjfcc\" (UID: \"92b024b5-d23c-4315-ad97-bed665cac395\") " pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:43.110780 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.110759    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92b024b5-d23c-4315-ad97-bed665cac395-must-gather-output\") pod \"must-gather-fjfcc\" (UID: \"92b024b5-d23c-4315-ad97-bed665cac395\") " pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:43.119057 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.119030    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8fv9\" (UniqueName: \"kubernetes.io/projected/92b024b5-d23c-4315-ad97-bed665cac395-kube-api-access-q8fv9\") pod \"must-gather-fjfcc\" (UID: \"92b024b5-d23c-4315-ad97-bed665cac395\") " pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:43.212057 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.211962    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n47qj/must-gather-fjfcc"
Jul 07 12:05:43.341396 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.341362    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n47qj/must-gather-fjfcc"]
Jul 07 12:05:43.344762 ip-10-0-143-227 kubenswrapper[2565]: W0707 12:05:43.344737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92b024b5_d23c_4315_ad97_bed665cac395.slice/crio-f3e6a1ede2971e5458d1f72b3d90794c64b0879a3ba5ec7e0873d7f8d4938e01 WatchSource:0}: Error finding container f3e6a1ede2971e5458d1f72b3d90794c64b0879a3ba5ec7e0873d7f8d4938e01: Status 404 returned error can't find the container with id f3e6a1ede2971e5458d1f72b3d90794c64b0879a3ba5ec7e0873d7f8d4938e01
Jul 07 12:05:43.346846 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.346829    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 12:05:43.489776 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:43.489687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n47qj/must-gather-fjfcc" event={"ID":"92b024b5-d23c-4315-ad97-bed665cac395","Type":"ContainerStarted","Data":"f3e6a1ede2971e5458d1f72b3d90794c64b0879a3ba5ec7e0873d7f8d4938e01"}
Jul 07 12:05:44.496409 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:44.496362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n47qj/must-gather-fjfcc" event={"ID":"92b024b5-d23c-4315-ad97-bed665cac395","Type":"ContainerStarted","Data":"248a0a0019a5ee7fcc43f300b2da8d30630d9c7fa8408a6fa847cd73e931f4a6"}
Jul 07 12:05:45.502031 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:45.501988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n47qj/must-gather-fjfcc" event={"ID":"92b024b5-d23c-4315-ad97-bed665cac395","Type":"ContainerStarted","Data":"e9dbf5417a8c743630f954c4409c629fefe492296b8d536af0323ed85544c6ad"}
Jul 07 12:05:45.520095 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:45.520043    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n47qj/must-gather-fjfcc" podStartSLOduration=2.533174691 podStartE2EDuration="3.520021935s" totalImagesPullingTime="986.847244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 12:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:05:45.518569213 +0000 UTC m=+1530.219502068" watchObservedRunningTime="2026-07-07 12:05:45.520021935 +0000 UTC m=+1530.220954795"
Jul 07 12:05:45.777551 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:45.777462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-85pvm_c996e346-c3a2-4e05-97c9-f3225a44bf73/global-pull-secret-syncer/0.log"
Jul 07 12:05:45.965194 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:45.965165    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pd5p5_7294718e-ce3b-429c-a703-826faaa72cf6/konnectivity-agent/0.log"
Jul 07 12:05:46.034684 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:46.034619    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-227.ec2.internal_e90e1b7edb9e4ce00addd9710b3030b5/haproxy/0.log"
Jul 07 12:05:49.700317 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:49.700265    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5nb98_74962eb8-f3f5-4ed0-9351-9ff787789d48/node-exporter/0.log"
Jul 07 12:05:49.730016 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:49.729981    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5nb98_74962eb8-f3f5-4ed0-9351-9ff787789d48/kube-rbac-proxy/0.log"
Jul 07 12:05:49.758626 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:49.758598    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5nb98_74962eb8-f3f5-4ed0-9351-9ff787789d48/init-textfile/0.log"
Jul 07 12:05:54.774851 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:54.774812    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l5mpr_1b7b5a55-0357-4094-b50c-528667fbf4b1/dns/0.log"
Jul 07 12:05:54.799741 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:54.799710    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l5mpr_1b7b5a55-0357-4094-b50c-528667fbf4b1/kube-rbac-proxy/0.log"
Jul 07 12:05:54.879669 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:54.879633    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vcqtg_aa3fdbfa-5535-4d08-a8f0-8052596a9a02/dns-node-resolver/0.log"
Jul 07 12:05:54.963352 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:54.963315    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"]
Jul 07 12:05:54.970841 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:54.970813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:54.973703 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:54.973678    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n47qj\"/\"default-dockercfg-jsv9v\""
Jul 07 12:05:54.982580 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:54.982550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"]
Jul 07 12:05:55.012910 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.012868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-lib-modules\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.013116 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.012949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-proc\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.013116 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.012994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-podres\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.013116 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.013042    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-sys\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.013116 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.013099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9df\" (UniqueName: \"kubernetes.io/projected/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-kube-api-access-bs9df\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114061 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-sys\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114061 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9df\" (UniqueName: \"kubernetes.io/projected/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-kube-api-access-bs9df\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114326 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-lib-modules\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114326 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-proc\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114326 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114206    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-podres\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114499 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114354    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-podres\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114499 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114414    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-sys\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114751 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114719    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-proc\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.114897 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.114728    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-lib-modules\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.124507 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.124447    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9df\" (UniqueName: \"kubernetes.io/projected/23d439ef-77f0-4bc4-833c-dfeb98a6dd1c-kube-api-access-bs9df\") pod \"perf-node-gather-daemonset-8psmc\" (UID: \"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c\") " pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.286144 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.286108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:55.409899 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.409860    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5h8h7_f0a1c866-f2cb-4c8d-805a-9ed813d9cbc4/node-ca/0.log"
Jul 07 12:05:55.445145 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.445116    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"]
Jul 07 12:05:55.448320 ip-10-0-143-227 kubenswrapper[2565]: W0707 12:05:55.448285    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23d439ef_77f0_4bc4_833c_dfeb98a6dd1c.slice/crio-27dbaa5c032403fd8bcebb7202bcf59f13003a9b4ca266d5334b5d924e5ad83e WatchSource:0}: Error finding container 27dbaa5c032403fd8bcebb7202bcf59f13003a9b4ca266d5334b5d924e5ad83e: Status 404 returned error can't find the container with id 27dbaa5c032403fd8bcebb7202bcf59f13003a9b4ca266d5334b5d924e5ad83e
Jul 07 12:05:55.557390 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:55.557359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc" event={"ID":"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c","Type":"ContainerStarted","Data":"27dbaa5c032403fd8bcebb7202bcf59f13003a9b4ca266d5334b5d924e5ad83e"}
Jul 07 12:05:56.563280 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:56.563238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc" event={"ID":"23d439ef-77f0-4bc4-833c-dfeb98a6dd1c","Type":"ContainerStarted","Data":"0df7c3b47e75476383a986443639871a33acd60931ff295dd514ad2dcda31c3a"}
Jul 07 12:05:56.563792 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:56.563361    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:05:56.581308 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:56.581251    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc" podStartSLOduration=2.581228889 podStartE2EDuration="2.581228889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 12:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 12:05:56.579868913 +0000 UTC m=+1541.280801772" watchObservedRunningTime="2026-07-07 12:05:56.581228889 +0000 UTC m=+1541.282161749"
Jul 07 12:05:56.682689 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:56.682661    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x6bk7_8b718c89-ec23-49c6-a70a-ebe38d5b482e/serve-healthcheck-canary/0.log"
Jul 07 12:05:57.095632 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:57.095602    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4qx5d_228dd7ca-ee07-481d-bd00-e8648e400bd2/kube-rbac-proxy/0.log"
Jul 07 12:05:57.121358 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:57.121326    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4qx5d_228dd7ca-ee07-481d-bd00-e8648e400bd2/exporter/0.log"
Jul 07 12:05:57.144368 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:57.144336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4qx5d_228dd7ca-ee07-481d-bd00-e8648e400bd2/extractor/0.log"
Jul 07 12:05:59.303117 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:59.303082    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-57676b4bd8-bm9sm_fc229ee2-e63f-45ad-a0a9-af51bd8e3a13/manager/0.log"
Jul 07 12:05:59.346403 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:59.346373    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-wn4v9_19f5c206-cf44-4e3e-9d9f-fe8fc7c052e8/server/0.log"
Jul 07 12:05:59.434762 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:59.434729    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-pkmcs_f1c18da9-65da-40e8-9364-b611023521f5/manager/0.log"
Jul 07 12:05:59.510023 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:05:59.509994    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-hm676_974faf44-7afd-4462-8a6d-77d2b0988ad7/seaweedfs/0.log"
Jul 07 12:06:02.579688 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:02.579657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n47qj/perf-node-gather-daemonset-8psmc"
Jul 07 12:06:04.529151 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.529112    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-697hj_ec46ad3a-d11e-416b-a447-c6625fe36c16/kube-multus/0.log"
Jul 07 12:06:04.554925 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.554899    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5rp_cc93da87-34c7-4bd9-a16b-dc777abfe795/kube-multus-additional-cni-plugins/0.log"
Jul 07 12:06:04.578735 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.578708    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5rp_cc93da87-34c7-4bd9-a16b-dc777abfe795/egress-router-binary-copy/0.log"
Jul 07 12:06:04.603223 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.603195    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5rp_cc93da87-34c7-4bd9-a16b-dc777abfe795/cni-plugins/0.log"
Jul 07 12:06:04.625382 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.625356    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5rp_cc93da87-34c7-4bd9-a16b-dc777abfe795/bond-cni-plugin/0.log"
Jul 07 12:06:04.648379 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.648351    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5rp_cc93da87-34c7-4bd9-a16b-dc777abfe795/routeoverride-cni/0.log"
Jul 07 12:06:04.671619 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.671521    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5rp_cc93da87-34c7-4bd9-a16b-dc777abfe795/whereabouts-cni-bincopy/0.log"
Jul 07 12:06:04.695007 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:04.694983    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-bv5rp_cc93da87-34c7-4bd9-a16b-dc777abfe795/whereabouts-cni/0.log"
Jul 07 12:06:05.218602 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:05.218573    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lrftd_6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2/network-metrics-daemon/0.log"
Jul 07 12:06:05.238379 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:05.238355    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lrftd_6c21f59a-0e3c-4fcb-a2f8-cfe9828c14c2/kube-rbac-proxy/0.log"
Jul 07 12:06:06.474809 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.474775    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-controller/0.log"
Jul 07 12:06:06.499293 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.499266    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/1.log"
Jul 07 12:06:06.499293 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.499272    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovn-acl-logging/0.log"
Jul 07 12:06:06.526866 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.526833    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/kube-rbac-proxy-node/0.log"
Jul 07 12:06:06.552847 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.552817    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 12:06:06.577801 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.577727    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/northd/0.log"
Jul 07 12:06:06.604427 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.604408    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/nbdb/0.log"
Jul 07 12:06:06.630576 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.630549    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/sbdb/0.log"
Jul 07 12:06:06.683548 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:06.683507    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-l5kdk_57472ab7-6f1d-4a9b-9149-bf550408b54a/ovnkube-controller/0.log"
Jul 07 12:06:07.676776 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:07.676740    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-82drw_074e34fb-c8fc-4531-a2f2-4978bdff12d0/network-check-target-container/0.log"
Jul 07 12:06:08.648431 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:08.648402    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jb4xk_4d4f48aa-03cb-46c7-8b68-210415f6f09f/iptables-alerter/0.log"
Jul 07 12:06:09.275036 ip-10-0-143-227 kubenswrapper[2565]: I0707 12:06:09.275007    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-sbkwj_d2bd7fa7-d91d-4cd3-8db8-2e2d3df39947/tuned/0.log"