Jul 13 20:19:02.410949 ip-10-0-131-42 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 20:19:02.839129 ip-10-0-131-42 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:19:02.839129 ip-10-0-131-42 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 20:19:02.839129 ip-10-0-131-42 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:19:02.839129 ip-10-0-131-42 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 20:19:02.839129 ip-10-0-131-42 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 20:19:02.839870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.839142    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 20:19:02.843236 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843218    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:02.843236 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843235    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843239    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843243    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843247    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843250    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843253    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843256    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843258    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843261    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843268    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843272    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843276    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843279    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843283    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843287    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843290    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843293    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843296    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843299    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:02.843311 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843301    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843304    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843307    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843310    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843314    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843317    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843321    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843326    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843329    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843332    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843335    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843338    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843341    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843344    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843346    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843349    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843352    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843355    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843358    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:02.843802 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843361    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843364    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843366    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843369    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843372    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843374    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843377    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843380    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843383    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843385    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843388    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843391    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843393    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843396    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843398    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843402    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843405    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843408    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843484    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843489    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:02.844286 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843492    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843496    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843499    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843503    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843506    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843509    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843512    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843516    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843518    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843523    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843527    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843530    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843533    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843537    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843540    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843543    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843546    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843548    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843551    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:02.844866 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843554    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843556    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843559    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843562    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843564    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843567    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843571    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843575    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843578    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843581    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843584    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843587    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843590    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843593    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843596    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843598    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843601    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843604    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843606    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843609    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:02.845346 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.843612    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844407    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844414    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844418    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844421    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844424    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844426    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844430    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844434    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844437    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844440    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844443    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844446    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844449    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844452    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844455    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844458    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844460    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844463    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844466    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:02.845873 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844469    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844472    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844475    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844477    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844480    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844484    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844487    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844489    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844492    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844495    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844498    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844500    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844505    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844509    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844513    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844516    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844519    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844522    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844525    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:02.846404 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844528    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844532    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844536    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844538    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844541    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844544    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844547    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844550    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844554    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844558    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844561    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844564    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844566    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844570    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844573    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844576    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844578    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844581    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844583    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844586    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:02.846911 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844589    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844592    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844595    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844597    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844600    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844603    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844605    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844608    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844611    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844614    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844616    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844619    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844622    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844625    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844628    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844631    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844634    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844639    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844643    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:02.847423 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844646    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844649    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844652    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844654    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844657    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844659    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844662    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844665    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844668    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844671    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844674    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844677    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844679    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844682    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844685    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844688    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844691    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844693    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844696    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844699    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:02.847926 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844701    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.844704    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845105    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845110    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845113    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845116    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845119    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845122    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845125    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845127    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845130    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845133    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845135    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845138    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845140    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845144    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845146    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845149    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845152    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845155    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:02.848446 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845157    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845160    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845163    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845166    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845169    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845172    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845175    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845178    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845181    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845183    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845186    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845189    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845191    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845194    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845197    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845200    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845202    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845205    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845207    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845210    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:02.848971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845213    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845216    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845219    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845221    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845224    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845227    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845229    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845232    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845235    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845238    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845241    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845244    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845246    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845249    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845253    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845256    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845258    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845262    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845264    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:02.849519 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845267    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845270    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845272    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845275    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845279    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845282    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845284    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845287    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845290    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845292    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845295    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845297    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845300    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845303    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845306    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845309    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845311    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845315    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845318    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845321    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:02.850033 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845327    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845330    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845333    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845335    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845339    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845341    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845344    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845349    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845352    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845355    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845357    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845363    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845367    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845370    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845373    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845377    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845381    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845384    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845387    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:02.850552 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845390    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845393    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.845397    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845481    2551 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845489    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845495    2551 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845499    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845503    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845507    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845512    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845516    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845520    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845524    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845529    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845532    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845536    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845539    2551 flags.go:64] FLAG: --cgroup-root=""
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845542    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845546    2551 flags.go:64] FLAG: --client-ca-file=""
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845549    2551 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845553    2551 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845558    2551 flags.go:64] FLAG: --cluster-domain=""
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845561    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845564    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 20:19:02.851048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845568    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845572    2551 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845576    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845579    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845583    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845586    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845589    2551 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845592    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845595    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845599    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845602    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845606    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845609    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845612    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845615    2551 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845619    2551 flags.go:64] FLAG: --enable-server="true"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845622    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845626    2551 flags.go:64] FLAG: --event-burst="100"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845630    2551 flags.go:64] FLAG: --event-qps="50"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845634    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845637    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845642    2551 flags.go:64] FLAG: --eviction-hard=""
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845646    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845649    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845652    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 20:19:02.851651 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845655    2551 flags.go:64] FLAG: --eviction-soft=""
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845659    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845661    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845666    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845669    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845672    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845675    2551 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845678    2551 flags.go:64] FLAG: --feature-gates=""
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845682    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845685    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845689    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845692    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845695    2551 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845699    2551 flags.go:64] FLAG: --help="false"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845702    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-131-42.ec2.internal"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845705    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845709    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845712    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845715    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845719    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845722    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845739    2551 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845742    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 20:19:02.852322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845745    2551 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845748    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845753    2551 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845756    2551 flags.go:64] FLAG: --kube-reserved=""
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845759    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845764    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845767    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845770    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845773    2551 flags.go:64] FLAG: --lock-file=""
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845776    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845780    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845783    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845790    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845793    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845796    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845800    2551 flags.go:64] FLAG: --logging-format="text"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845803    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845807    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845810    2551 flags.go:64] FLAG: --manifest-url=""
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845813    2551 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845821    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845824    2551 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845829    2551 flags.go:64] FLAG: --max-pods="110"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845832    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845836    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 20:19:02.853219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845839    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845842    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845845    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845848    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845851    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845859    2551 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845862    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845865    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845869    2551 flags.go:64] FLAG: --pod-cidr=""
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845872    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845879    2551 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845882    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845886    2551 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845891    2551 flags.go:64] FLAG: --port="10250"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845894    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845897    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-040f860318d3bba6f"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845901    2551 flags.go:64] FLAG: --qos-reserved=""
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845904    2551 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845908    2551 flags.go:64] FLAG: --register-node="true"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845912    2551 flags.go:64] FLAG: --register-with-taints=""
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845916    2551 flags.go:64] FLAG: --registry-burst="10"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845919    2551 flags.go:64] FLAG: --registry-qps="5"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845922    2551 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845925    2551 flags.go:64] FLAG: --reserved-memory=""
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845929    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 20:19:02.854183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845933    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845936    2551 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845939    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845943    2551 flags.go:64] FLAG: --runonce="false"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845947    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845950    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845953    2551 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845956    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845960    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845963    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845967    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845970    2551 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845973    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845976    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845979    2551 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845982    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845985    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845989    2551 flags.go:64] FLAG: --system-cgroups=""
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845992    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.845998    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846002    2551 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846005    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846009    2551 flags.go:64] FLAG: --tls-min-version=""
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846012    2551 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846015    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 20:19:02.854879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846019    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846023    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846027    2551 flags.go:64] FLAG: --v="2"
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846031    2551 flags.go:64] FLAG: --version="false"
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846035    2551 flags.go:64] FLAG: --vmodule=""
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846039    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846043    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846173    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846178    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846182    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846185    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846188    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846191    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846194    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846197    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846200    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846204    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846208    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846211    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846214    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846217    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:02.855526 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846220    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846222    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846225    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846228    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846231    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846234    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846238    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846242    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846246    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846249    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846252    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846255    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846259    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846261    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846264    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846267    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846270    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846272    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846275    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:02.856088 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846278    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846280    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846283    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846287    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846291    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846294    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846297    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846300    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846303    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846306    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846309    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846312    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846315    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846318    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846321    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846324    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846327    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846330    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846332    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846719    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:02.856582 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846722    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846737    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846740    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846743    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846746    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846750    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846753    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846756    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846759    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846761    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846764    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846767    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846770    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846773    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846776    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846779    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846782    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846784    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846787    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846790    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:02.857261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846793    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846795    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846798    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846801    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846804    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846807    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846809    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846813    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846816    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846818    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846821    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846826    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846829    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846832    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846835    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846838    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846840    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846845    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846847    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:02.857809 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846850    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846853    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846855    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846858    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846861    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846864    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.846867    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.846873    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.855120    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.855138    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.856014    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.856023    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856089    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856096    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856099    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:02.858302 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856103    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856106    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856110    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856113    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856116    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856119    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856122    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856125    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856128    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856131    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856134    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856137    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856140    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856143    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856146    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856149    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856152    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856155    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856157    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856160    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:02.858718 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856163    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856166    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856169    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856172    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856175    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856178    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856182    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856184    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856187    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856190    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856193    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856195    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856198    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856201    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856204    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856206    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856209    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856213    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856217    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:02.859261 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856221    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856225    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856228    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856231    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856235    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856238    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856241    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856244    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856247    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856250    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856254    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856258    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856261    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856264    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856267    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856270    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856274    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856277    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856280    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856285    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:02.859889 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856289    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856293    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856296    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856299    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856302    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856305    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856308    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856311    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856314    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856317    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856320    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856322    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856325    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856328    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856331    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856334    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856336    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856340    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856343    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856346    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:02.860451 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856349    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856352    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856355    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856358    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856360    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856363    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856366    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856369    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856372    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856375    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856378    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856381    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856384    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856387    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856390    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856393    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:02.860971 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856396    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.856401    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856523    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856530    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856533    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856536    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856539    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856543    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856545    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856548    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856551    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856554    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856557    2551 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856560    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856564    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 20:19:02.861396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856567    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856570    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856572    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856576    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856579    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856581    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856584    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856587    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856590    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856594    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856597    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856600    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856603    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856606    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856608    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856611    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856615    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856617    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856620    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856623    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 20:19:02.861822 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856626    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856629    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856631    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856634    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856637    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856640    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856642    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856645    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856648    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856650    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856653    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856655    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856659    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856662    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856665    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856667    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856670    2551 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856673    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856675    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856678    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 20:19:02.862326 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856681    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856684    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856687    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856690    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856692    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856695    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856698    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856701    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856703    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856706    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856709    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856712    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856714    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856717    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856720    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856723    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856744    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856749    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856752    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 20:19:02.862875 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856756    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856760    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856763    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856766    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856770    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856773    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856777    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856780    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856783    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856785    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856788    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856791    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856793    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856796    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856799    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856803    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856806    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856810    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856812    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 20:19:02.863392 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856815    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856818    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856821    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856823    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856826    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856829    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856831    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:02.856834    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.856840    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.856964    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.861372    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.862815    2551 server.go:1019] "Starting client certificate rotation"
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.862919    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 20:19:02.863948 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.862958    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 20:19:02.865515 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.865500    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 20:19:02.866740 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.866718    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 20:19:02.875898 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.875879    2551 log.go:25] "Validated CRI v1 runtime API"
Jul 13 20:19:02.878057 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.878042    2551 log.go:25] "Validated CRI v1 image API"
Jul 13 20:19:02.878960 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.878944    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 20:19:02.883683 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.883655    2551 fs.go:135] Filesystem UUIDs: map[2b45495a-e83c-4ddd-ae75-6007bff7c1c0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 939c5028-85d2-41c8-8695-efad3bf61154:/dev/nvme0n1p4]
Jul 13 20:19:02.883759 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.883683    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 20:19:02.891710 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.891588    2551 manager.go:216] Machine: {Timestamp:2026-07-13 20:19:02.89013788 +0000 UTC m=+0.362366648 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097041 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea876e23945dcbf8e4c3a2c0abd7a SystemUUID:ec2ea876-e239-45dc-bf8e-4c3a2c0abd7a BootID:c3780b99-3b6b-4a8a-a57a-4ce5d3a9d7c6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:04:e8:b4:19:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:04:e8:b4:19:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:b7:ff:3b:0c:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 20:19:02.891710 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.891701    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 20:19:02.891863 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.891805    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 20:19:02.895354 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.895326    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 20:19:02.895499 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.895358    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-42.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 20:19:02.895800 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.895784    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 20:19:02.896148 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.896136    2551 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 20:19:02.896183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.896151    2551 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 20:19:02.896183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.896159    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 20:19:02.896183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.896176    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 20:19:02.897467 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.897456    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 20:19:02.900248 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900235    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 20:19:02.900407 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900397    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 20:19:02.900449 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900428    2551 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 20:19:02.900449 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900438    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 20:19:02.900522 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900460    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 20:19:02.900522 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900471    2551 kubelet.go:398] "Adding apiserver pod source"
Jul 13 20:19:02.900522 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900485    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 20:19:02.900939 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900925    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 20:19:02.900999 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.900949    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 20:19:02.903746 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.903718    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 20:19:02.905707 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905692    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 20:19:02.905775 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905740    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 20:19:02.905918 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905906    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 20:19:02.905961 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905930    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 20:19:02.905961 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905945    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 20:19:02.905961 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905959    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 20:19:02.906062 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905978    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 20:19:02.906062 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.905992    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 20:19:02.906494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.906483    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 20:19:02.906494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.906495    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 20:19:02.906554 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.906502    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 20:19:02.906554 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.906508    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 20:19:02.906554 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.906517    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 20:19:02.906554 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.906526    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 20:19:02.907462 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.907449    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 20:19:02.907462 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.907463    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 20:19:02.910914 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.910899    2551 server.go:1268] "Started kubelet"
Jul 13 20:19:02.911101 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.911039    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 20:19:02.911163 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.911092    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 20:19:02.911212 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.911174    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 20:19:02.912417 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.912399    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 20:19:02.912762 ip-10-0-131-42 systemd[1]: Started Kubernetes Kubelet.
Jul 13 20:19:02.914056 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.914033    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-42.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 20:19:02.914135 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.914110    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 20:19:02.914214 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.914189    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 20:19:02.915292 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.915278    2551 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 20:19:02.916167 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.916139    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 20:19:02.918870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.918849    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 20:19:02.918870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.918860    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 20:19:02.919364 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.919002    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:02.919592 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919575    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 20:19:02.919592 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919588    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 20:19:02.919740 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919605    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 20:19:02.919818 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919749    2551 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 20:19:02.919818 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919759    2551 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 20:19:02.919952 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919937    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 20:19:02.920046 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919955    2551 factory.go:55] Registering systemd factory
Jul 13 20:19:02.920046 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.919965    2551 factory.go:223] Registration of the systemd container factory successfully
Jul 13 20:19:02.920198 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.920182    2551 factory.go:153] Registering CRI-O factory
Jul 13 20:19:02.920252 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.920204    2551 factory.go:223] Registration of the crio container factory successfully
Jul 13 20:19:02.920252 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.920225    2551 factory.go:103] Registering Raw factory
Jul 13 20:19:02.920252 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.920234    2551 manager.go:1196] Started watching for new ooms in manager
Jul 13 20:19:02.920464 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.919321    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-42.ec2.internal.18c1f2d963c16707  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-42.ec2.internal,UID:ip-10-0-131-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-42.ec2.internal,},FirstTimestamp:2026-07-13 20:19:02.910871303 +0000 UTC m=+0.383100070,LastTimestamp:2026-07-13 20:19:02.910871303 +0000 UTC m=+0.383100070,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-42.ec2.internal,}"
Jul 13 20:19:02.920756 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.920739    2551 manager.go:349] Starting recovery of all containers
Jul 13 20:19:02.922694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.922665    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p2k87"
Jul 13 20:19:02.929749 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.929691    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-42.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 20:19:02.930026 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.929996    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 20:19:02.935451 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.935433    2551 manager.go:354] Recovery completed
Jul 13 20:19:02.936675 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.936656    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p2k87"
Jul 13 20:19:02.941425 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.941408    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:02.943914 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.943898    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:02.943991 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.943927    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:02.943991 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.943937    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:02.944534 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.944518    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 20:19:02.944534 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.944532    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:02.944619 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.944534    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 20:19:02.944619 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.944615    2551 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 20:19:02.945325 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.945309    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:02.945375 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.945337    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:02.945375 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.945349    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:02.945652 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.945640    2551 policy_none.go:49] "None policy: Start"
Jul 13 20:19:02.945694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.945657    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 20:19:02.945694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.945668    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 20:19:02.946467 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.946402    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-42.ec2.internal.18c1f2d965b9949f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-42.ec2.internal,UID:ip-10-0-131-42.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-42.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-42.ec2.internal,},FirstTimestamp:2026-07-13 20:19:02.943913119 +0000 UTC m=+0.416141889,LastTimestamp:2026-07-13 20:19:02.943913119 +0000 UTC m=+0.416141889,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-42.ec2.internal,}"
Jul 13 20:19:02.946715 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.946702    2551 policy_none.go:47] "Start"
Jul 13 20:19:02.993352 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.993329    2551 manager.go:337] "Starting Device Plugin manager"
Jul 13 20:19:02.993487 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.993379    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 20:19:02.993487 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.993391    2551 server.go:87] "Starting device plugin registration server"
Jul 13 20:19:02.993727 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.993715    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 20:19:02.993777 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.993731    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 20:19:02.993968 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.993944    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 20:19:02.994208 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.994168    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 20:19:02.994208 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:02.994198    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 20:19:02.994521 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.994485    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 20:19:02.994623 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:02.994546    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.094252 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.094181    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:03.095410 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.095344    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:03.095410 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.095377    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:03.095410 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.095387    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:03.095546 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.095417    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.103347 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.103330    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.103404 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.103354    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-42.ec2.internal\": node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.116855 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.116827    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.129389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.129362    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 20:19:03.130777 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.130756    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 20:19:03.130871 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.130791    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 20:19:03.130871 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.130812    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 20:19:03.130871 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.130844    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 20:19:03.133418 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.133400    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:03.221407 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.221355    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.231274 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.231242    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal"]
Jul 13 20:19:03.231346 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.231322    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:03.233194 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.233176    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:03.233256 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.233217    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:03.233256 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.233232    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:03.234548 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.234535    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:03.234747 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.234729    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.234801 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.234767    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:03.235447 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.235423    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:03.235447 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.235452    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:03.235599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.235462    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:03.235599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.235424    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:03.235599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.235535    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:03.235599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.235546    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:03.236764 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.236749    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.236821 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.236777    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 20:19:03.237494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.237473    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 20:19:03.237592 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.237503    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 20:19:03.237592 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.237518    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeHasSufficientPID"
Jul 13 20:19:03.264567 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.264544    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-42.ec2.internal\" not found" node="ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.269162 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.269144    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-42.ec2.internal\" not found" node="ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.322037 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.322007    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.333931 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.333705    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/836a542284f56721e298e3c6c6747941-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal\" (UID: \"836a542284f56721e298e3c6c6747941\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.333931 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.333754    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a542284f56721e298e3c6c6747941-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal\" (UID: \"836a542284f56721e298e3c6c6747941\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.333931 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.333800    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08333fc818b35b3d22fa5e3da933609-config\") pod \"kube-apiserver-proxy-ip-10-0-131-42.ec2.internal\" (UID: \"e08333fc818b35b3d22fa5e3da933609\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.423199 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.423098    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.434962 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.434939    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/836a542284f56721e298e3c6c6747941-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal\" (UID: \"836a542284f56721e298e3c6c6747941\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.435096 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.434972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a542284f56721e298e3c6c6747941-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal\" (UID: \"836a542284f56721e298e3c6c6747941\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.435096 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.434992    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08333fc818b35b3d22fa5e3da933609-config\") pod \"kube-apiserver-proxy-ip-10-0-131-42.ec2.internal\" (UID: \"e08333fc818b35b3d22fa5e3da933609\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.435096 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.435035    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e08333fc818b35b3d22fa5e3da933609-config\") pod \"kube-apiserver-proxy-ip-10-0-131-42.ec2.internal\" (UID: \"e08333fc818b35b3d22fa5e3da933609\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.435096 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.435070    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/836a542284f56721e298e3c6c6747941-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal\" (UID: \"836a542284f56721e298e3c6c6747941\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.435096 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.435082    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836a542284f56721e298e3c6c6747941-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal\" (UID: \"836a542284f56721e298e3c6c6747941\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.526058 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.526010    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.568438 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.568411    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.571541 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.571524    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:03.626291 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.626237    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.727053 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.726966    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.780802 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.780774    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:03.827294 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.827255    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.863217 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.863184    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 20:19:03.863716 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.863367    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 20:19:03.863716 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.863390    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 20:19:03.927965 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:03.927936    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:03.940208 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.940174    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 20:14:03 +0000 UTC" deadline="2028-02-22 10:46:06.350176198 +0000 UTC"
Jul 13 20:19:03.940273 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:03.940208    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14126h27m2.409974874s"
Jul 13 20:19:04.028268 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.028180    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:04.089285 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.089257    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:04.128743 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.128694    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:04.189633 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:04.189600    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08333fc818b35b3d22fa5e3da933609.slice/crio-0c44baca79204738a51965f9ce04b9b4bd736b7e13df6e0d5eabf42ad64bf099 WatchSource:0}: Error finding container 0c44baca79204738a51965f9ce04b9b4bd736b7e13df6e0d5eabf42ad64bf099: Status 404 returned error can't find the container with id 0c44baca79204738a51965f9ce04b9b4bd736b7e13df6e0d5eabf42ad64bf099
Jul 13 20:19:04.195189 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.195170    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:19:04.228799 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.228764    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:04.281733 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:04.281702    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836a542284f56721e298e3c6c6747941.slice/crio-65f13bdb95088fcba31026e9472b58c44e1938018955f67232b54d0be09633a8 WatchSource:0}: Error finding container 65f13bdb95088fcba31026e9472b58c44e1938018955f67232b54d0be09633a8: Status 404 returned error can't find the container with id 65f13bdb95088fcba31026e9472b58c44e1938018955f67232b54d0be09633a8
Jul 13 20:19:04.329202 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.329160    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:04.429922 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.429895    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-42.ec2.internal\" not found"
Jul 13 20:19:04.455104 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.455082    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:04.521688 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.521644    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:04.535613 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.535558    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 20:19:04.536605 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.536592    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal"
Jul 13 20:19:04.548040 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.548020    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 20:19:04.904143 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.904046    2551 apiserver.go:52] "Watching apiserver"
Jul 13 20:19:04.910202 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.910178    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 20:19:04.911635 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.911608    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-5hd22","openshift-multus/multus-additional-cni-plugins-5crvv","openshift-multus/network-metrics-daemon-6wlbd","kube-system/global-pull-secret-syncer-vjwb9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal","openshift-multus/multus-95p9h","openshift-network-diagnostics/network-check-target-xqc68","openshift-network-operator/iptables-alerter-mlkqq","openshift-ovn-kubernetes/ovnkube-node-zmz6m","kube-system/konnectivity-agent-8xhr5","kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h","openshift-cluster-node-tuning-operator/tuned-zfhc6","openshift-dns/node-resolver-js24l"]
Jul 13 20:19:04.913253 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.913230    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:04.914583 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.914563    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:04.915932 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.915912    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 20:19:04.916187 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.916166    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:04.916306 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.916238    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:04.916710 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.916691    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t6pwq\""
Jul 13 20:19:04.917494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.916876    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 20:19:04.917494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.916943    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 20:19:04.917494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.917158    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 20:19:04.917494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.917309    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mpbn4\""
Jul 13 20:19:04.917494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.917312    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:04.917494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.917428    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 20:19:04.917494 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.917479    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:04.919489 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.919462    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 20:19:04.920303 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.920117    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.921698 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.921677    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:04.922669 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.922307    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 20:19:04.922669 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.922320    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 20:19:04.922669 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.922510    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ncvtf\""
Jul 13 20:19:04.923176 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.922733    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 20:19:04.923176 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.922797    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 20:19:04.923497 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.923388    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:04.924785 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.924146    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 20:19:04.924785 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.924481    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 20:19:04.928292 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.926043    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:04.928292 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.927364    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 20:19:04.928292 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.927997    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 20:19:04.928292 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.928200    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 20:19:04.928553 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.928417    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 20:19:04.928553 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.928417    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 20:19:04.928553 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.928517    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 20:19:04.929300 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.929272    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kwp7q\""
Jul 13 20:19:04.929300 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.929288    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7npsl\""
Jul 13 20:19:04.929864 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.929843    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:04.933140 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.932701    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:04.933550 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.933529    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 20:19:04.933907 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.933886    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 20:19:04.934060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.934044    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 20:19:04.934176 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.934156    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 20:19:04.934275 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.934114    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 20:19:04.934491 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.934468    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jnv4n\""
Jul 13 20:19:04.934689 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.934671    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dhntx\""
Jul 13 20:19:04.934775 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.934700    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:04.934833 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:04.934782    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:04.935234 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.935206    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 20:19:04.935327 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.935306    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 20:19:04.935555 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.935530    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xtrqn\""
Jul 13 20:19:04.936400 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.936146    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:04.938028 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.938009    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 20:19:04.938423 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.938390    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 20:19:04.938692 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.938657    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8xl26\""
Jul 13 20:19:04.938812 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.938790    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 20:19:04.942717 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.942696    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 20:19:04.944539 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944471    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:04.944539 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944506    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xj4\" (UniqueName: \"kubernetes.io/projected/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-kube-api-access-f5xj4\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:04.944539 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944536    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-os-release\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.944781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944559    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-cni-multus\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.944781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944585    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-etc-kubernetes\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.944781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944609    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3531f22-c8c0-440e-a943-81e2490fc57d-konnectivity-ca\") pod \"konnectivity-agent-8xhr5\" (UID: \"a3531f22-c8c0-440e-a943-81e2490fc57d\") " pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:04.944781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944633    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxxj\" (UniqueName: \"kubernetes.io/projected/22830676-1f0c-40d7-928d-7809f15bc5db-kube-api-access-7pxxj\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:04.944781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944658    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:04.944781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-cni-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.944781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944771    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-k8s-cni-cncf-io\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944804    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-netns\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944867    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-conf-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944906    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22830676-1f0c-40d7-928d-7809f15bc5db-iptables-alerter-script\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.944953    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-system-cni-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945003    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-cnibin\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945029    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-socket-dir-parent\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945054    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-daemon-config\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945097    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-multus-certs\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945128    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3531f22-c8c0-440e-a943-81e2490fc57d-agent-certs\") pod \"konnectivity-agent-8xhr5\" (UID: \"a3531f22-c8c0-440e-a943-81e2490fc57d\") " pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:04.945190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945155    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22830676-1f0c-40d7-928d-7809f15bc5db-host-slash\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:04.945602 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945263    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-cni-binary-copy\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945602 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945302    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-cni-bin\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945602 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945332    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-kubelet\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945602 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945372    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-hostroot\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.945602 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.945408    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcn4\" (UniqueName: \"kubernetes.io/projected/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-kube-api-access-rrcn4\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:04.957770 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.957748    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:04.965653 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.965628    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z7kjg"
Jul 13 20:19:04.983312 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:04.983288    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z7kjg"
Jul 13 20:19:05.021315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.021282    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 20:19:05.046090 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046054    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-etc-selinux\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.046255 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046100    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-log-socket\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.046255 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046176    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-slash\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.046255 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046208    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-socket-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.046255 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-sys\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046265    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxxj\" (UniqueName: \"kubernetes.io/projected/22830676-1f0c-40d7-928d-7809f15bc5db-kube-api-access-7pxxj\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-k8s-cni-cncf-io\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046336    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-host\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046358    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36e04abd-ca7e-44c7-ab7b-441feed30455-tmp\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046389    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-system-cni-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046445    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-k8s-cni-cncf-io\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046463    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-system-cni-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.046479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046451    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046500    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8607e642-2424-4480-bef1-ec3bb89574ed-ovn-node-metrics-cert\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3531f22-c8c0-440e-a943-81e2490fc57d-agent-certs\") pod \"konnectivity-agent-8xhr5\" (UID: \"a3531f22-c8c0-440e-a943-81e2490fc57d\") " pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046552    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-hostroot\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046578    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmglm\" (UniqueName: \"kubernetes.io/projected/095468f0-3e04-4916-a8eb-c0119d910b11-kube-api-access-dmglm\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046629    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-hostroot\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046685    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-etc-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2066f3e0-e084-4802-8e22-bf7eec308a63-tmp-dir\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-lib-modules\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046825    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.046881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046866    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-ovn\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.047392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046893    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2066f3e0-e084-4802-8e22-bf7eec308a63-hosts-file\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.047392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046920    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-sys-fs\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.047392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.046965    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-var-lib-kubelet\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.047392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047003    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-cni-bin\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.047392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047053    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-ovnkube-script-lib\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.047392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047089    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 20:19:05.047392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047114    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnsxr\" (UniqueName: \"kubernetes.io/projected/8607e642-2424-4480-bef1-ec3bb89574ed-kube-api-access-xnsxr\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.047810 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047156    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.047810 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047732    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c02c555-f585-4b0f-877d-766cecd345e7-serviceca\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.047810 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047776    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-device-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.047810 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047803    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-modprobe-d\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047827    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysctl-d\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047851    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-systemd\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047881    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcn4\" (UniqueName: \"kubernetes.io/projected/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-kube-api-access-rrcn4\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047909    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-registration-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047934    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysconfig\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047962    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-systemd\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.047985    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-var-lib-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.048024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048016    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-cni-netd\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048042    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-systemd-units\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048071    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048101    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xj4\" (UniqueName: \"kubernetes.io/projected/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-kube-api-access-f5xj4\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048127    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-os-release\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048157    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-cni-multus\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048285    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-cni-multus\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.048401 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048284    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.048782 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.048438    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:05.048782 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-os-release\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.048782 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048521    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:05.048782 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.048589    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs podName:bc4b6b4f-5b06-4ace-8cef-2964b68d10fb nodeName:}" failed. No retries permitted until 2026-07-13 20:19:05.548561746 +0000 UTC m=+3.020790519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs") pod "network-metrics-daemon-6wlbd" (UID: "bc4b6b4f-5b06-4ace-8cef-2964b68d10fb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:05.048782 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048651    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-netns\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048800    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf83bc3-42e1-409e-a511-bce2009ba373-kubelet-config\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048834    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-env-overrides\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048886    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-netns\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048868    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbc2\" (UniqueName: \"kubernetes.io/projected/de209486-217b-4fb2-bb81-fb4f185312c1-kube-api-access-clbc2\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048938    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-daemon-config\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048967    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-multus-certs\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.048999    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysctl-conf\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049024    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-kubelet\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.049052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049052    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22830676-1f0c-40d7-928d-7809f15bc5db-host-slash\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049064    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-run-multus-certs\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049079    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-cni-binary-copy\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049106    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-kubernetes\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049129    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-run\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049138    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22830676-1f0c-40d7-928d-7809f15bc5db-host-slash\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049156    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-node-log\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049181    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-system-cni-dir\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049209    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-os-release\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049258    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049290    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-etc-kubernetes\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049329    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf83bc3-42e1-409e-a511-bce2009ba373-dbus\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049389    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-etc-kubernetes\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.049475 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-run-netns\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049478    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049520    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049547    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwbz\" (UniqueName: \"kubernetes.io/projected/2066f3e0-e084-4802-8e22-bf7eec308a63-kube-api-access-9lwbz\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049589    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3531f22-c8c0-440e-a943-81e2490fc57d-konnectivity-ca\") pod \"konnectivity-agent-8xhr5\" (UID: \"a3531f22-c8c0-440e-a943-81e2490fc57d\") " pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049616    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-cni-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049643    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-conf-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049694    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-ovnkube-config\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049719    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-cnibin\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049772    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22830676-1f0c-40d7-928d-7809f15bc5db-iptables-alerter-script\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049798    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-cnibin\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049789    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-cni-binary-copy\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049827    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-socket-dir-parent\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049883    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-tuned\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049803    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-daemon-config\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.049886    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-cni-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050313    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-conf-dir\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050394    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46wc\" (UniqueName: \"kubernetes.io/projected/36e04abd-ca7e-44c7-ab7b-441feed30455-kube-api-access-l46wc\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050426    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c02c555-f585-4b0f-877d-766cecd345e7-host\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050483    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4qz\" (UniqueName: \"kubernetes.io/projected/7c02c555-f585-4b0f-877d-766cecd345e7-kube-api-access-5b4qz\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050555    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-cni-bin\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050590    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-kubelet\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050749    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-kubelet\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.050880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050848    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-host-var-lib-cni-bin\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.051289 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050891    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3531f22-c8c0-440e-a943-81e2490fc57d-konnectivity-ca\") pod \"konnectivity-agent-8xhr5\" (UID: \"a3531f22-c8c0-440e-a943-81e2490fc57d\") " pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:05.051289 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.050912    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-multus-socket-dir-parent\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.051289 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.051016    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-cnibin\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.051434 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.051366    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22830676-1f0c-40d7-928d-7809f15bc5db-iptables-alerter-script\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:05.052009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.051970    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3531f22-c8c0-440e-a943-81e2490fc57d-agent-certs\") pod \"konnectivity-agent-8xhr5\" (UID: \"a3531f22-c8c0-440e-a943-81e2490fc57d\") " pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:05.056516 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.056495    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:05.056634 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.056520    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:05.056634 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.056535    2551 projected.go:196] Error preparing data for projected volume kube-api-access-vwd8w for pod openshift-network-diagnostics/network-check-target-xqc68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:05.056634 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.056598    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w podName:2771d026-f54c-4c67-a476-d3f5742dab82 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:05.556581403 +0000 UTC m=+3.028810184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vwd8w" (UniqueName: "kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w") pod "network-check-target-xqc68" (UID: "2771d026-f54c-4c67-a476-d3f5742dab82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:05.058949 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.058928    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xj4\" (UniqueName: \"kubernetes.io/projected/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-kube-api-access-f5xj4\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:05.059269 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.059245    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxxj\" (UniqueName: \"kubernetes.io/projected/22830676-1f0c-40d7-928d-7809f15bc5db-kube-api-access-7pxxj\") pod \"iptables-alerter-mlkqq\" (UID: \"22830676-1f0c-40d7-928d-7809f15bc5db\") " pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:05.059840 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.059819    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcn4\" (UniqueName: \"kubernetes.io/projected/23205ad9-b80f-49b6-9a92-4aeb0654f0e8-kube-api-access-rrcn4\") pod \"multus-95p9h\" (UID: \"23205ad9-b80f-49b6-9a92-4aeb0654f0e8\") " pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.135859 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.135795    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal" event={"ID":"836a542284f56721e298e3c6c6747941","Type":"ContainerStarted","Data":"65f13bdb95088fcba31026e9472b58c44e1938018955f67232b54d0be09633a8"}
Jul 13 20:19:05.137059 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.137032    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal" event={"ID":"e08333fc818b35b3d22fa5e3da933609","Type":"ContainerStarted","Data":"0c44baca79204738a51965f9ce04b9b4bd736b7e13df6e0d5eabf42ad64bf099"}
Jul 13 20:19:05.151668 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151630    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-etc-selinux\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.151822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151682    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-log-socket\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.151822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151708    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-slash\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.151822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151732    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-socket-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.151822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151804    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-log-socket\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151824    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-etc-selinux\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151810    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-slash\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151838    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-sys\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151881    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-socket-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-sys\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151899    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-host\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151937    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36e04abd-ca7e-44c7-ab7b-441feed30455-tmp\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151963    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151977    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-host\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.151994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.151989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8607e642-2424-4480-bef1-ec3bb89574ed-ovn-node-metrics-cert\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152018    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmglm\" (UniqueName: \"kubernetes.io/projected/095468f0-3e04-4916-a8eb-c0119d910b11-kube-api-access-dmglm\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.152450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152032    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152043    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-etc-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152068    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2066f3e0-e084-4802-8e22-bf7eec308a63-tmp-dir\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.152450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152316    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-etc-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152330    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2066f3e0-e084-4802-8e22-bf7eec308a63-tmp-dir\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.152450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152429    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-lib-modules\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152483    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-ovn\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152498    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2066f3e0-e084-4802-8e22-bf7eec308a63-hosts-file\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152545    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2066f3e0-e084-4802-8e22-bf7eec308a63-hosts-file\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152575    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-ovn\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.152580    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152606    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-sys-fs\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152624    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-lib-modules\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.152662    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret podName:5bf83bc3-42e1-409e-a511-bce2009ba373 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:05.652639685 +0000 UTC m=+3.124868460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret") pod "global-pull-secret-syncer-vjwb9" (UID: "5bf83bc3-42e1-409e-a511-bce2009ba373") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-var-lib-kubelet\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152715    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-cni-bin\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152713    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-sys-fs\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152741    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-ovnkube-script-lib\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152768    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnsxr\" (UniqueName: \"kubernetes.io/projected/8607e642-2424-4480-bef1-ec3bb89574ed-kube-api-access-xnsxr\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152783    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-var-lib-kubelet\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152795    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.152811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-cni-bin\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152822    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c02c555-f585-4b0f-877d-766cecd345e7-serviceca\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152850    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-device-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152878    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-modprobe-d\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152903    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysctl-d\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152927    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-systemd\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152973    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-registration-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.152997    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysconfig\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153022    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-systemd\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153047    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-var-lib-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-cni-netd\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153096    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-systemd-units\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153141    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysctl-d\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153147    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153196    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153197    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153248    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.153706 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153251    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysconfig\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153300    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-registration-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153306    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf83bc3-42e1-409e-a511-bce2009ba373-kubelet-config\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153329    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-run-systemd\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153334    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c02c555-f585-4b0f-877d-766cecd345e7-serviceca\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153356    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-var-lib-openvswitch\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153338    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-env-overrides\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153384    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-cni-netd\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153395    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbc2\" (UniqueName: \"kubernetes.io/projected/de209486-217b-4fb2-bb81-fb4f185312c1-kube-api-access-clbc2\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153409    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-systemd-units\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153438    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-systemd\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153460    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysctl-conf\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-kubelet\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153482    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-ovnkube-script-lib\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153499    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-kubernetes\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153462    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153516    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-run\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.154622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153542    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-run\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153555    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-kubelet\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153577    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bf83bc3-42e1-409e-a511-bce2009ba373-kubelet-config\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153600    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-node-log\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153619    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-system-cni-dir\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153642    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-system-cni-dir\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153651    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-modprobe-d\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153660    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-sysctl-conf\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153674    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/095468f0-3e04-4916-a8eb-c0119d910b11-device-dir\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153666    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-node-log\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-env-overrides\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153749    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-os-release\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-kubernetes\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153793    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153803    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-os-release\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153836    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf83bc3-42e1-409e-a511-bce2009ba373-dbus\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153871    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-run-netns\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.155562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153896    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153923    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153947    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwbz\" (UniqueName: \"kubernetes.io/projected/2066f3e0-e084-4802-8e22-bf7eec308a63-kube-api-access-9lwbz\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.153978    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-ovnkube-config\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154006    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-cnibin\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154032    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154074    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-tuned\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154099    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46wc\" (UniqueName: \"kubernetes.io/projected/36e04abd-ca7e-44c7-ab7b-441feed30455-kube-api-access-l46wc\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154123    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c02c555-f585-4b0f-877d-766cecd345e7-host\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154142    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4qz\" (UniqueName: \"kubernetes.io/projected/7c02c555-f585-4b0f-877d-766cecd345e7-kube-api-access-5b4qz\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154289    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154717    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-run-ovn-kubernetes\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154792    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/de209486-217b-4fb2-bb81-fb4f185312c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154801    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8607e642-2424-4480-bef1-ec3bb89574ed-host-run-netns\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154822    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8607e642-2424-4480-bef1-ec3bb89574ed-ovnkube-config\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154860    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c02c555-f585-4b0f-877d-766cecd345e7-host\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154877    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-cnibin\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.156547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154931    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de209486-217b-4fb2-bb81-fb4f185312c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.157470 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.154935    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bf83bc3-42e1-409e-a511-bce2009ba373-dbus\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.157470 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.155178    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8607e642-2424-4480-bef1-ec3bb89574ed-ovn-node-metrics-cert\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.157470 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.155928    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36e04abd-ca7e-44c7-ab7b-441feed30455-tmp\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.157470 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.157034    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/36e04abd-ca7e-44c7-ab7b-441feed30455-etc-tuned\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.165085 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.165055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmglm\" (UniqueName: \"kubernetes.io/projected/095468f0-3e04-4916-a8eb-c0119d910b11-kube-api-access-dmglm\") pod \"aws-ebs-csi-driver-node-85b6h\" (UID: \"095468f0-3e04-4916-a8eb-c0119d910b11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.168561 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.168507    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46wc\" (UniqueName: \"kubernetes.io/projected/36e04abd-ca7e-44c7-ab7b-441feed30455-kube-api-access-l46wc\") pod \"tuned-zfhc6\" (UID: \"36e04abd-ca7e-44c7-ab7b-441feed30455\") " pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.168561 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.168519    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4qz\" (UniqueName: \"kubernetes.io/projected/7c02c555-f585-4b0f-877d-766cecd345e7-kube-api-access-5b4qz\") pod \"node-ca-5hd22\" (UID: \"7c02c555-f585-4b0f-877d-766cecd345e7\") " pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.168755 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.168740    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnsxr\" (UniqueName: \"kubernetes.io/projected/8607e642-2424-4480-bef1-ec3bb89574ed-kube-api-access-xnsxr\") pod \"ovnkube-node-zmz6m\" (UID: \"8607e642-2424-4480-bef1-ec3bb89574ed\") " pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.169814 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.169792    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwbz\" (UniqueName: \"kubernetes.io/projected/2066f3e0-e084-4802-8e22-bf7eec308a63-kube-api-access-9lwbz\") pod \"node-resolver-js24l\" (UID: \"2066f3e0-e084-4802-8e22-bf7eec308a63\") " pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.170086 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.170067    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbc2\" (UniqueName: \"kubernetes.io/projected/de209486-217b-4fb2-bb81-fb4f185312c1-kube-api-access-clbc2\") pod \"multus-additional-cni-plugins-5crvv\" (UID: \"de209486-217b-4fb2-bb81-fb4f185312c1\") " pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.231006 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.230960    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:05.241150 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.241114    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mlkqq"
Jul 13 20:19:05.251112 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.251085    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-95p9h"
Jul 13 20:19:05.258106 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.258082    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:05.266813 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.266788    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zfhc6"
Jul 13 20:19:05.275700 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.275678    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5crvv"
Jul 13 20:19:05.282565 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.282540    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5hd22"
Jul 13 20:19:05.290394 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.290362    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-js24l"
Jul 13 20:19:05.296419 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.296385    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h"
Jul 13 20:19:05.557616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.557532    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:05.557616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.557577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:05.557872 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.557731    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:05.557872 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.557746    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:05.557872 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.557765    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:05.557872 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.557777    2551 projected.go:196] Error preparing data for projected volume kube-api-access-vwd8w for pod openshift-network-diagnostics/network-check-target-xqc68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:05.557872 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.557812    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs podName:bc4b6b4f-5b06-4ace-8cef-2964b68d10fb nodeName:}" failed. No retries permitted until 2026-07-13 20:19:06.557790569 +0000 UTC m=+4.030019328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs") pod "network-metrics-daemon-6wlbd" (UID: "bc4b6b4f-5b06-4ace-8cef-2964b68d10fb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:05.557872 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.557835    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w podName:2771d026-f54c-4c67-a476-d3f5742dab82 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:06.557822859 +0000 UTC m=+4.030051616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwd8w" (UniqueName: "kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w") pod "network-check-target-xqc68" (UID: "2771d026-f54c-4c67-a476-d3f5742dab82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:05.658043 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.658001    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:05.658209 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.658165    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:05.658267 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:05.658243    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret podName:5bf83bc3-42e1-409e-a511-bce2009ba373 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:06.658226406 +0000 UTC m=+4.130455164 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret") pod "global-pull-secret-syncer-vjwb9" (UID: "5bf83bc3-42e1-409e-a511-bce2009ba373") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:05.808244 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.808204    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c02c555_f585_4b0f_877d_766cecd345e7.slice/crio-2ecf7cbfd8edd64ea2d097e8b4443a08c831f7dfa5cd1e71889e7594a521a6cb WatchSource:0}: Error finding container 2ecf7cbfd8edd64ea2d097e8b4443a08c831f7dfa5cd1e71889e7594a521a6cb: Status 404 returned error can't find the container with id 2ecf7cbfd8edd64ea2d097e8b4443a08c831f7dfa5cd1e71889e7594a521a6cb
Jul 13 20:19:05.811729 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.811684    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3531f22_c8c0_440e_a943_81e2490fc57d.slice/crio-d67706298cf95d2dbefc786ba6b82b9494700ad5cae2ca81f2b8ad4c3f2cc6d4 WatchSource:0}: Error finding container d67706298cf95d2dbefc786ba6b82b9494700ad5cae2ca81f2b8ad4c3f2cc6d4: Status 404 returned error can't find the container with id d67706298cf95d2dbefc786ba6b82b9494700ad5cae2ca81f2b8ad4c3f2cc6d4
Jul 13 20:19:05.812387 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.812349    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22830676_1f0c_40d7_928d_7809f15bc5db.slice/crio-b26ef5038ab37624273e7a1840d76c31ab7ba972b0b3fcc36b92e023aa78e816 WatchSource:0}: Error finding container b26ef5038ab37624273e7a1840d76c31ab7ba972b0b3fcc36b92e023aa78e816: Status 404 returned error can't find the container with id b26ef5038ab37624273e7a1840d76c31ab7ba972b0b3fcc36b92e023aa78e816
Jul 13 20:19:05.813487 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.813462    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095468f0_3e04_4916_a8eb_c0119d910b11.slice/crio-f6f6dc75f930aeaae4d377e19f057351d40211e44048de4faa72ee9a971a367e WatchSource:0}: Error finding container f6f6dc75f930aeaae4d377e19f057351d40211e44048de4faa72ee9a971a367e: Status 404 returned error can't find the container with id f6f6dc75f930aeaae4d377e19f057351d40211e44048de4faa72ee9a971a367e
Jul 13 20:19:05.814881 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.814767    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23205ad9_b80f_49b6_9a92_4aeb0654f0e8.slice/crio-4081713020f5e19647db39c82c0d7aa403adb235ec5bd6a70848494fddcff452 WatchSource:0}: Error finding container 4081713020f5e19647db39c82c0d7aa403adb235ec5bd6a70848494fddcff452: Status 404 returned error can't find the container with id 4081713020f5e19647db39c82c0d7aa403adb235ec5bd6a70848494fddcff452
Jul 13 20:19:05.816489 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.815772    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde209486_217b_4fb2_bb81_fb4f185312c1.slice/crio-185f2a16a224b521f46b046ed5fe013459f77da11e46e1b23c517fd395430563 WatchSource:0}: Error finding container 185f2a16a224b521f46b046ed5fe013459f77da11e46e1b23c517fd395430563: Status 404 returned error can't find the container with id 185f2a16a224b521f46b046ed5fe013459f77da11e46e1b23c517fd395430563
Jul 13 20:19:05.817977 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.817933    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2066f3e0_e084_4802_8e22_bf7eec308a63.slice/crio-5361b41d371f6c3918e2ebb56fac6697e327f0107179ef133609ec8516e22c6f WatchSource:0}: Error finding container 5361b41d371f6c3918e2ebb56fac6697e327f0107179ef133609ec8516e22c6f: Status 404 returned error can't find the container with id 5361b41d371f6c3918e2ebb56fac6697e327f0107179ef133609ec8516e22c6f
Jul 13 20:19:05.819128 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:05.819101    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8607e642_2424_4480_bef1_ec3bb89574ed.slice/crio-3bbbfd0eb0f6a62ef74d1ff00640efe44c6f9f0e7de7228501bad6bcad4f0ff6 WatchSource:0}: Error finding container 3bbbfd0eb0f6a62ef74d1ff00640efe44c6f9f0e7de7228501bad6bcad4f0ff6: Status 404 returned error can't find the container with id 3bbbfd0eb0f6a62ef74d1ff00640efe44c6f9f0e7de7228501bad6bcad4f0ff6
Jul 13 20:19:05.986051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.985979    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:14:05 +0000 UTC" deadline="2028-04-03 13:36:56.960516068 +0000 UTC"
Jul 13 20:19:05.986051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:05.986044    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15113h17m50.974475084s"
Jul 13 20:19:06.071463 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.071382    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 20:19:06.139808 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.139770    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5hd22" event={"ID":"7c02c555-f585-4b0f-877d-766cecd345e7","Type":"ContainerStarted","Data":"2ecf7cbfd8edd64ea2d097e8b4443a08c831f7dfa5cd1e71889e7594a521a6cb"}
Jul 13 20:19:06.140881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.140852    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zfhc6" event={"ID":"36e04abd-ca7e-44c7-ab7b-441feed30455","Type":"ContainerStarted","Data":"53b8cebb72093e9b0040fb09a9207782a24cf60223bf134b7bae457bab3959ab"}
Jul 13 20:19:06.142060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.142038    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"3bbbfd0eb0f6a62ef74d1ff00640efe44c6f9f0e7de7228501bad6bcad4f0ff6"}
Jul 13 20:19:06.143483 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.143286    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-js24l" event={"ID":"2066f3e0-e084-4802-8e22-bf7eec308a63","Type":"ContainerStarted","Data":"5361b41d371f6c3918e2ebb56fac6697e327f0107179ef133609ec8516e22c6f"}
Jul 13 20:19:06.144427 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.144399    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerStarted","Data":"185f2a16a224b521f46b046ed5fe013459f77da11e46e1b23c517fd395430563"}
Jul 13 20:19:06.145488 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.145460    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h" event={"ID":"095468f0-3e04-4916-a8eb-c0119d910b11","Type":"ContainerStarted","Data":"f6f6dc75f930aeaae4d377e19f057351d40211e44048de4faa72ee9a971a367e"}
Jul 13 20:19:06.146470 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.146442    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8xhr5" event={"ID":"a3531f22-c8c0-440e-a943-81e2490fc57d","Type":"ContainerStarted","Data":"d67706298cf95d2dbefc786ba6b82b9494700ad5cae2ca81f2b8ad4c3f2cc6d4"}
Jul 13 20:19:06.148203 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.148184    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal" event={"ID":"e08333fc818b35b3d22fa5e3da933609","Type":"ContainerStarted","Data":"d2d0da7fb1f11d4f55588600a02ab38e5a8b3111ee96ba1c86d8cb2974931ccb"}
Jul 13 20:19:06.149248 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.149228    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95p9h" event={"ID":"23205ad9-b80f-49b6-9a92-4aeb0654f0e8","Type":"ContainerStarted","Data":"4081713020f5e19647db39c82c0d7aa403adb235ec5bd6a70848494fddcff452"}
Jul 13 20:19:06.150248 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.150229    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mlkqq" event={"ID":"22830676-1f0c-40d7-928d-7809f15bc5db","Type":"ContainerStarted","Data":"b26ef5038ab37624273e7a1840d76c31ab7ba972b0b3fcc36b92e023aa78e816"}
Jul 13 20:19:06.167318 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.167276    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-42.ec2.internal" podStartSLOduration=1.16726501 podStartE2EDuration="1.16726501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:06.167183681 +0000 UTC m=+3.639412458" watchObservedRunningTime="2026-07-13 20:19:06.16726501 +0000 UTC m=+3.639493787"
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.565103    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.565156    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.565310    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.565333    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.565346    2551 projected.go:196] Error preparing data for projected volume kube-api-access-vwd8w for pod openshift-network-diagnostics/network-check-target-xqc68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.565407    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w podName:2771d026-f54c-4c67-a476-d3f5742dab82 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:08.565388382 +0000 UTC m=+6.037617140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwd8w" (UniqueName: "kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w") pod "network-check-target-xqc68" (UID: "2771d026-f54c-4c67-a476-d3f5742dab82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.565487    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:06.565951 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.565532    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs podName:bc4b6b4f-5b06-4ace-8cef-2964b68d10fb nodeName:}" failed. No retries permitted until 2026-07-13 20:19:08.565511647 +0000 UTC m=+6.037740406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs") pod "network-metrics-daemon-6wlbd" (UID: "bc4b6b4f-5b06-4ace-8cef-2964b68d10fb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:06.666624 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.665936    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:06.666624 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.666126    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:06.666624 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:06.666191    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret podName:5bf83bc3-42e1-409e-a511-bce2009ba373 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:08.666171889 +0000 UTC m=+6.138400648 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret") pod "global-pull-secret-syncer-vjwb9" (UID: "5bf83bc3-42e1-409e-a511-bce2009ba373") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:06.986491 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.986445    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 20:14:05 +0000 UTC" deadline="2028-01-11 10:24:47.856792801 +0000 UTC"
Jul 13 20:19:06.986491 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:06.986490    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13118h5m40.870307559s"
Jul 13 20:19:07.133849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:07.132920    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:07.133849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:07.132974    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:07.133849 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:07.133071    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:07.133849 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:07.133564    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:07.133849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:07.133639    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:07.133849 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:07.133760    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:08.585327 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:08.585285    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:08.585874 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:08.585344    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:08.585874 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.585537    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:08.585874 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.585559    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:08.585874 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.585573    2551 projected.go:196] Error preparing data for projected volume kube-api-access-vwd8w for pod openshift-network-diagnostics/network-check-target-xqc68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:08.585874 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.585640    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w podName:2771d026-f54c-4c67-a476-d3f5742dab82 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:12.585617732 +0000 UTC m=+10.057846488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwd8w" (UniqueName: "kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w") pod "network-check-target-xqc68" (UID: "2771d026-f54c-4c67-a476-d3f5742dab82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:08.586219 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.586088    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:08.586219 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.586147    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs podName:bc4b6b4f-5b06-4ace-8cef-2964b68d10fb nodeName:}" failed. No retries permitted until 2026-07-13 20:19:12.586128724 +0000 UTC m=+10.058357480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs") pod "network-metrics-daemon-6wlbd" (UID: "bc4b6b4f-5b06-4ace-8cef-2964b68d10fb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:08.686352 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:08.686307    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:08.686559 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.686533    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:08.686629 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:08.686602    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret podName:5bf83bc3-42e1-409e-a511-bce2009ba373 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:12.686581902 +0000 UTC m=+10.158810663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret") pod "global-pull-secret-syncer-vjwb9" (UID: "5bf83bc3-42e1-409e-a511-bce2009ba373") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:09.133118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:09.132393    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:09.133118 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:09.132554    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:09.133118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:09.132873    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:09.133118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:09.132918    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:09.133118 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:09.133019    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:09.133550 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:09.133195    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:11.132682 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:11.131968    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:11.132682 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:11.132121    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:11.132682 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:11.132341    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:11.132682 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:11.132450    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:11.132682 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:11.132508    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:11.132682 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:11.132600    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:12.625290 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:12.624995    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:12.625290 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:12.625128    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:12.625290 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.625252    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:12.625290 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.625273    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:12.626001 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.625306    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:12.626001 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.625321    2551 projected.go:196] Error preparing data for projected volume kube-api-access-vwd8w for pod openshift-network-diagnostics/network-check-target-xqc68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:12.626001 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.625327    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs podName:bc4b6b4f-5b06-4ace-8cef-2964b68d10fb nodeName:}" failed. No retries permitted until 2026-07-13 20:19:20.625306053 +0000 UTC m=+18.097534807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs") pod "network-metrics-daemon-6wlbd" (UID: "bc4b6b4f-5b06-4ace-8cef-2964b68d10fb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:12.626001 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.625378    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w podName:2771d026-f54c-4c67-a476-d3f5742dab82 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:20.625359534 +0000 UTC m=+18.097588303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwd8w" (UniqueName: "kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w") pod "network-check-target-xqc68" (UID: "2771d026-f54c-4c67-a476-d3f5742dab82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:12.726910 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:12.726261    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:12.726910 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.726467    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:12.726910 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:12.726535    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret podName:5bf83bc3-42e1-409e-a511-bce2009ba373 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:20.726514925 +0000 UTC m=+18.198743697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret") pod "global-pull-secret-syncer-vjwb9" (UID: "5bf83bc3-42e1-409e-a511-bce2009ba373") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:13.132512 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:13.132172    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:13.132512 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:13.132264    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:13.132512 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:13.132309    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:13.132512 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:13.132306    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:13.132512 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:13.132387    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:13.132512 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:13.132474    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:15.134564 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:15.133769    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:15.134564 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:15.133968    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:15.134564 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:15.134321    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:15.134564 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:15.134411    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:15.134564 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:15.134450    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:15.134564 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:15.134515    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:17.131572 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:17.131503    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:17.132036 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:17.131502    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:17.132036 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:17.131639    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:17.132036 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:17.131677    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:17.132036 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:17.131779    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:17.132036 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:17.131874    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:19.131831 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:19.131796    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:19.132279 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:19.131845    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:19.132279 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:19.131811    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:19.132279 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:19.131934    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:19.132279 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:19.132034    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:19.132279 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:19.132138    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:20.683994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:20.683948    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:20.683994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:20.684010    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:20.684548 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.684123    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:20.684548 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.684162    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:20.684548 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.684180    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:20.684548 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.684192    2551 projected.go:196] Error preparing data for projected volume kube-api-access-vwd8w for pod openshift-network-diagnostics/network-check-target-xqc68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:20.684548 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.684201    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs podName:bc4b6b4f-5b06-4ace-8cef-2964b68d10fb nodeName:}" failed. No retries permitted until 2026-07-13 20:19:36.684178406 +0000 UTC m=+34.156407160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs") pod "network-metrics-daemon-6wlbd" (UID: "bc4b6b4f-5b06-4ace-8cef-2964b68d10fb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:20.684548 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.684243    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w podName:2771d026-f54c-4c67-a476-d3f5742dab82 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:36.684227431 +0000 UTC m=+34.156456188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwd8w" (UniqueName: "kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w") pod "network-check-target-xqc68" (UID: "2771d026-f54c-4c67-a476-d3f5742dab82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:20.784929 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:20.784887    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:20.785082 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.785020    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:20.785153 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:20.785091    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret podName:5bf83bc3-42e1-409e-a511-bce2009ba373 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:36.785070639 +0000 UTC m=+34.257299394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret") pod "global-pull-secret-syncer-vjwb9" (UID: "5bf83bc3-42e1-409e-a511-bce2009ba373") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:21.131208 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:21.131131    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:21.131370 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:21.131136    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:21.131370 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:21.131244    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:21.131370 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:21.131346    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:21.131511 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:21.131372    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:21.131511 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:21.131469    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:23.132278 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:23.132239    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:23.132743 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:23.132340    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:23.132743 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:23.132357    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:23.132743 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:23.132437    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:23.132743 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:23.132477    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:23.132743 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:23.132546    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:25.131962 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:25.131925    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:25.132397 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:25.131923    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:25.132397 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:25.132059    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:25.132397 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:25.132151    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:25.132397 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:25.131927    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:25.132397 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:25.132237    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:26.203146 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.202701    2551 generic.go:356] "Generic (PLEG): container finished" podID="836a542284f56721e298e3c6c6747941" containerID="c92e0142125ba04e1aec751417d818799d2e81006acf597dc5dd3bcc3435c184" exitCode=0
Jul 13 20:19:26.203935 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.202766    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal" event={"ID":"836a542284f56721e298e3c6c6747941","Type":"ContainerDied","Data":"c92e0142125ba04e1aec751417d818799d2e81006acf597dc5dd3bcc3435c184"}
Jul 13 20:19:26.205615 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.205588    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:19:26.205956 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.205930    2551 generic.go:356] "Generic (PLEG): container finished" podID="8607e642-2424-4480-bef1-ec3bb89574ed" containerID="b745af25e414cc01ccdc176ad87047e2344cfb31a1c9cbe0ce69df51312acd27" exitCode=1
Jul 13 20:19:26.206142 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.205955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"ae5d67d07254fc42dda43ec0b6f0e4ed847b92e58658ef43c50dee20e8d8c3fa"}
Jul 13 20:19:26.206142 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.206016    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"b8a76d0b357d20609537b42ef4b1e697d0f0ada2cff4d913109b65877626283f"}
Jul 13 20:19:26.206142 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.206109    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"ca446870db966a3e10d254b5aa4f09ad4c813ef6aeec0a9009b054c9c74870d2"}
Jul 13 20:19:26.206310 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.206161    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerDied","Data":"b745af25e414cc01ccdc176ad87047e2344cfb31a1c9cbe0ce69df51312acd27"}
Jul 13 20:19:26.206310 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.206205    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"37cb231bdb603ac29e0a715e258237d7f5e7fa2d098d966930b1e4814879d021"}
Jul 13 20:19:26.207444 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.207419    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-js24l" event={"ID":"2066f3e0-e084-4802-8e22-bf7eec308a63","Type":"ContainerStarted","Data":"b364f070d747ad8e78171de9a4070eac7fb22dc49d940c0f5e23db040db07223"}
Jul 13 20:19:26.208839 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.208818    2551 generic.go:356] "Generic (PLEG): container finished" podID="de209486-217b-4fb2-bb81-fb4f185312c1" containerID="eea77a6279ab708f720679f823bc3bce2b4e33c719f7ba6d9d3ed02932d224ad" exitCode=0
Jul 13 20:19:26.208930 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.208896    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerDied","Data":"eea77a6279ab708f720679f823bc3bce2b4e33c719f7ba6d9d3ed02932d224ad"}
Jul 13 20:19:26.210511 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.210495    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h" event={"ID":"095468f0-3e04-4916-a8eb-c0119d910b11","Type":"ContainerStarted","Data":"c769bb69740f614d3effab4c18f09af2daae884f88958e05c0ac6296e3b3eeb5"}
Jul 13 20:19:26.213963 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.213933    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8xhr5" event={"ID":"a3531f22-c8c0-440e-a943-81e2490fc57d","Type":"ContainerStarted","Data":"07d78765e50e073514db604d1dfb3596e737614d6d4b12d1e35654279729d1fe"}
Jul 13 20:19:26.215388 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.215360    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-95p9h" event={"ID":"23205ad9-b80f-49b6-9a92-4aeb0654f0e8","Type":"ContainerStarted","Data":"98aad2e57f99e2b8738677f0f99c7a6698fc3a5c0ffb93924161d3953ef73bce"}
Jul 13 20:19:26.216801 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.216777    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5hd22" event={"ID":"7c02c555-f585-4b0f-877d-766cecd345e7","Type":"ContainerStarted","Data":"fdcc95b2d39323c7e5320fe057fa698baf54fb2957eba8eaccf687d56f26c646"}
Jul 13 20:19:26.218101 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.218078    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zfhc6" event={"ID":"36e04abd-ca7e-44c7-ab7b-441feed30455","Type":"ContainerStarted","Data":"52a5b19ea94497be5b15287d73c01ab2efea6c03ec5e0ef39e635213e272b275"}
Jul 13 20:19:26.250486 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.250443    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8xhr5" podStartSLOduration=7.195230028 podStartE2EDuration="23.250429504s" totalImagesPullingTime="16.055199476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:26.234698852 +0000 UTC m=+23.706927626" watchObservedRunningTime="2026-07-13 20:19:26.250429504 +0000 UTC m=+23.722658326"
Jul 13 20:19:26.250597 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.250578    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zfhc6" podStartSLOduration=3.519941627 podStartE2EDuration="23.250574837s" totalImagesPullingTime="19.73063321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:26.250202827 +0000 UTC m=+23.722431601" watchObservedRunningTime="2026-07-13 20:19:26.250574837 +0000 UTC m=+23.722803610"
Jul 13 20:19:26.264873 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.264830    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-js24l" podStartSLOduration=3.569577851 podStartE2EDuration="23.264815097s" totalImagesPullingTime="19.695237246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:26.264682988 +0000 UTC m=+23.736911763" watchObservedRunningTime="2026-07-13 20:19:26.264815097 +0000 UTC m=+23.737043871"
Jul 13 20:19:26.281745 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.281636    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-95p9h" podStartSLOduration=3.5217947069999997 podStartE2EDuration="23.281623419s" totalImagesPullingTime="19.759828712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:26.281267983 +0000 UTC m=+23.753496756" watchObservedRunningTime="2026-07-13 20:19:26.281623419 +0000 UTC m=+23.753852216"
Jul 13 20:19:26.305048 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.305025    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:26.305731 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.305713    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:26.334457 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:26.334404    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5hd22" podStartSLOduration=3.65349538 podStartE2EDuration="23.334383643s" totalImagesPullingTime="19.680888263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:26.317169069 +0000 UTC m=+23.789397845" watchObservedRunningTime="2026-07-13 20:19:26.334383643 +0000 UTC m=+23.806612418"
Jul 13 20:19:27.131854 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.131825    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:27.131956 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.131858    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:27.132003 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.131957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:27.132003 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:27.131956    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:27.132117 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:27.132094    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:27.132206 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:27.132189    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:27.181362 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.181336    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 20:19:27.221182 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.221153    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mlkqq" event={"ID":"22830676-1f0c-40d7-928d-7809f15bc5db","Type":"ContainerStarted","Data":"09007f9ae6d7c7400431b543fc6ffeeb3e5eed3037ebc6a59703bbc14e16d45e"}
Jul 13 20:19:27.222954 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.222928    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal" event={"ID":"836a542284f56721e298e3c6c6747941","Type":"ContainerStarted","Data":"6dadbfda7a097c9afd849ee0174d803330d29c2da22e11a1fb3d781c5df191d0"}
Jul 13 20:19:27.225601 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.225585    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:19:27.226067 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.226045    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"ba475ccc33cbb5983f2bdf53e7a76fcf09ac3b1f52bff60039f138efe2146f63"}
Jul 13 20:19:27.227699 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.227682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h" event={"ID":"095468f0-3e04-4916-a8eb-c0119d910b11","Type":"ContainerStarted","Data":"f5aabc509ade4cd4d65119564524c61d3a5c9317e7bd646dc65626fa6d9866a3"}
Jul 13 20:19:27.228067 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.228049    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:27.228283 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.228262    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8xhr5"
Jul 13 20:19:27.240559 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.240522    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mlkqq" podStartSLOduration=4.5301593570000005 podStartE2EDuration="24.240509993s" totalImagesPullingTime="19.710350636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:27.239830192 +0000 UTC m=+24.712058966" watchObservedRunningTime="2026-07-13 20:19:27.240509993 +0000 UTC m=+24.712738767"
Jul 13 20:19:27.255517 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:27.255478    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-42.ec2.internal" podStartSLOduration=22.255467398 podStartE2EDuration="22.255467398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:27.255255678 +0000 UTC m=+24.727484451" watchObservedRunningTime="2026-07-13 20:19:27.255467398 +0000 UTC m=+24.727696171"
Jul 13 20:19:28.012347 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:28.012248    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T20:19:27.181358282Z","UUID":"18fa88d7-22b4-4d0a-8c41-71bd4bfad1c7","Handler":null,"Name":"","Endpoint":""}
Jul 13 20:19:28.014359 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:28.014336    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 20:19:28.014487 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:28.014377    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 20:19:29.104738 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.104695    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:29.131039 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.131010    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:29.131220 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.131010    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:29.131220 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:29.131133    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:29.131346 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.131010    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:29.131346 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:29.131270    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:29.131448 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:29.131378    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:29.204838 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.204800    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:29.238740 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.238484    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:19:29.239173 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.239143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"ac498b8bb91bfb3ec909558a95769b49b2d3b024789002fd277f84ebdd62a781"}
Jul 13 20:19:29.241369 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.241344    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h" event={"ID":"095468f0-3e04-4916-a8eb-c0119d910b11","Type":"ContainerStarted","Data":"31a30a7e41c5989a7e2bc8735e6356bd703b7145289145196861292125427380"}
Jul 13 20:19:29.263455 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:29.263403    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-85b6h" podStartSLOduration=3.309305861 podStartE2EDuration="25.263388631s" totalImagesPullingTime="21.95408277s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:29.263039177 +0000 UTC m=+26.735267963" watchObservedRunningTime="2026-07-13 20:19:29.263388631 +0000 UTC m=+26.735617405"
Jul 13 20:19:30.489035 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:30.488996    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:31.131852 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.131818    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:31.132011 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.131818    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:31.132011 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:31.131931    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:31.132011 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.131818    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:31.132110 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:31.132005    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:31.132110 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:31.132071    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:31.250462 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.250182    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:19:31.250827 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.250496    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"2fea1704d8397499c3c79db90e73e7fe4a4f486818a437696853dfb1a1c97f9a"}
Jul 13 20:19:31.251500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.250948    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:31.251500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.251029    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:31.251500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.251031    2551 scope.go:117] "RemoveContainer" containerID="b745af25e414cc01ccdc176ad87047e2344cfb31a1c9cbe0ce69df51312acd27"
Jul 13 20:19:31.251500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.251078    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:31.269307 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.269237    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:31.270071 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:31.269999    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:19:32.256459 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:32.256282    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:19:32.257207 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:32.256839    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" event={"ID":"8607e642-2424-4480-bef1-ec3bb89574ed","Type":"ContainerStarted","Data":"850c434d55a7cc294b36dc17a1fca8d5e46d66f9c360c2aef48603a38a917547"}
Jul 13 20:19:32.258645 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:32.258619    2551 generic.go:356] "Generic (PLEG): container finished" podID="de209486-217b-4fb2-bb81-fb4f185312c1" containerID="b0c2b7d2d3bd6f288ae4f748bb3dfb4fa208c2fab51c232538000f9849758eb8" exitCode=0
Jul 13 20:19:32.258763 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:32.258676    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerDied","Data":"b0c2b7d2d3bd6f288ae4f748bb3dfb4fa208c2fab51c232538000f9849758eb8"}
Jul 13 20:19:32.324558 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:32.324508    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m" podStartSLOduration=9.554210529 podStartE2EDuration="29.324492397s" totalImagesPullingTime="19.770281868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:32.293915935 +0000 UTC m=+29.766144722" watchObservedRunningTime="2026-07-13 20:19:32.324492397 +0000 UTC m=+29.796721171"
Jul 13 20:19:33.132590 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.132552    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:33.132801 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.132668    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:33.132801 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:33.132671    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:33.132801 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:33.132782    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:33.132923 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.132811    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:33.132923 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:33.132876    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:33.595220 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.595177    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjwb9"]
Jul 13 20:19:33.595600 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.595358    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:33.595600 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:33.595461    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:33.595813 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.595797    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqc68"]
Jul 13 20:19:33.595880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.595871    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:33.595946 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:33.595931    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:33.598554 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.598532    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wlbd"]
Jul 13 20:19:33.598681 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.598618    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:33.598817 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:33.598715    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:33.910777 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:33.910745    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:34.011459 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:34.011390    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:34.111795 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:34.111762    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:34.264623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:34.264591    2551 generic.go:356] "Generic (PLEG): container finished" podID="de209486-217b-4fb2-bb81-fb4f185312c1" containerID="dac7d14f7e12df464069637af4b8d599a946deba815537ba7f41fbdfc358ef78" exitCode=0
Jul 13 20:19:34.264778 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:34.264639    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerDied","Data":"dac7d14f7e12df464069637af4b8d599a946deba815537ba7f41fbdfc358ef78"}
Jul 13 20:19:34.610345 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:34.610318    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:34.673484 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:34.673444    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:35.131209 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:35.131180    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:35.131364 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:35.131189    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:35.131364 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:35.131276    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:35.131462 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:35.131395    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:35.131462 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:35.131451    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:35.131570 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:35.131551    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:35.416904 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:35.416706    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:35.517188 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:35.517152    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:36.272066 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:36.272031    2551 generic.go:356] "Generic (PLEG): container finished" podID="de209486-217b-4fb2-bb81-fb4f185312c1" containerID="874a47acfac8c90f7446196df814c0546d467f66e549fce7fa077340094f2752" exitCode=0
Jul 13 20:19:36.272449 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:36.272089    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerDied","Data":"874a47acfac8c90f7446196df814c0546d467f66e549fce7fa077340094f2752"}
Jul 13 20:19:36.697266 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:36.697222    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:36.697461 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:36.697277    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:36.697461 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.697404    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:36.697461 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.697428    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 20:19:36.697461 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.697444    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 20:19:36.697461 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.697454    2551 projected.go:196] Error preparing data for projected volume kube-api-access-vwd8w for pod openshift-network-diagnostics/network-check-target-xqc68: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:36.697645 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.697481    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs podName:bc4b6b4f-5b06-4ace-8cef-2964b68d10fb nodeName:}" failed. No retries permitted until 2026-07-13 20:20:08.697465013 +0000 UTC m=+66.169693776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs") pod "network-metrics-daemon-6wlbd" (UID: "bc4b6b4f-5b06-4ace-8cef-2964b68d10fb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 20:19:36.697645 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.697499    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w podName:2771d026-f54c-4c67-a476-d3f5742dab82 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:08.697491182 +0000 UTC m=+66.169719934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwd8w" (UniqueName: "kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w") pod "network-check-target-xqc68" (UID: "2771d026-f54c-4c67-a476-d3f5742dab82") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 20:19:36.798246 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:36.798203    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:36.798435 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.798406    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:36.798492 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:36.798484    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret podName:5bf83bc3-42e1-409e-a511-bce2009ba373 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:08.798465277 +0000 UTC m=+66.270694032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret") pod "global-pull-secret-syncer-vjwb9" (UID: "5bf83bc3-42e1-409e-a511-bce2009ba373") : object "kube-system"/"original-pull-secret" not registered
Jul 13 20:19:37.131622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:37.131536    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:37.131622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:37.131572    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:37.131862 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:37.131687    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:37.131862 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:37.131709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:37.131862 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:37.131807    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:37.132009 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:37.131899    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:39.131967 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.131928    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:39.132380 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.131981    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:39.132380 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.131928    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:39.132380 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.132082    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wlbd" podUID="bc4b6b4f-5b06-4ace-8cef-2964b68d10fb"
Jul 13 20:19:39.132380 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.132164    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjwb9" podUID="5bf83bc3-42e1-409e-a511-bce2009ba373"
Jul 13 20:19:39.132380 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.132251    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xqc68" podUID="2771d026-f54c-4c67-a476-d3f5742dab82"
Jul 13 20:19:39.394616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.394578    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-42.ec2.internal" event="NodeReady"
Jul 13 20:19:39.394811 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.394765    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 20:19:39.447821 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.447791    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79bd95c96d-zwg5j"]
Jul 13 20:19:39.452510 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.452489    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.456510 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.456486    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 20:19:39.456640 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.456562    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 20:19:39.456640 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.456572    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cj8kp\""
Jul 13 20:19:39.456884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.456861    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 20:19:39.464360 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.464340    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fwtrx"]
Jul 13 20:19:39.467710 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.467687    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lqt7x"]
Jul 13 20:19:39.469726 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.469697    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:39.472216 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.472185    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 20:19:39.472880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.472845    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79bd95c96d-zwg5j"]
Jul 13 20:19:39.472985 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.472941    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.472985 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.472976    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 20:19:39.473197 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.473177    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6zs7s\""
Jul 13 20:19:39.473902 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.473736    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 20:19:39.474528 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.474508    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 20:19:39.475665 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.475632    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26pr8\""
Jul 13 20:19:39.475768 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.475643    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 20:19:39.476538 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.476468    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 20:19:39.477687 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.477166    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fwtrx"]
Jul 13 20:19:39.501032 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.501003    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqt7x"]
Jul 13 20:19:39.620402 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620367    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b05a9784-a678-478a-9694-00408b02e4cf-cert\") pod \"ingress-canary-fwtrx\" (UID: \"b05a9784-a678-478a-9694-00408b02e4cf\") " pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:39.620402 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620411    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-certificates\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.620676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620443    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gsds\" (UniqueName: \"kubernetes.io/projected/b05a9784-a678-478a-9694-00408b02e4cf-kube-api-access-9gsds\") pod \"ingress-canary-fwtrx\" (UID: \"b05a9784-a678-478a-9694-00408b02e4cf\") " pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:39.620676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620474    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36de539f-0164-4957-bb2d-e422fd60ea8e-ca-trust-extracted\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.620676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620495    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-bound-sa-token\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.620676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620546    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-image-registry-private-configuration\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.620676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620572    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.620676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620588    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5rw\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-kube-api-access-mf5rw\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.620676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620643    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-installation-pull-secrets\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.621009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b92591-8c5b-4fb5-ba4b-271efe211e11-tmp-dir\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.621009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620741    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17b92591-8c5b-4fb5-ba4b-271efe211e11-config-volume\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.621009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620764    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcvn\" (UniqueName: \"kubernetes.io/projected/17b92591-8c5b-4fb5-ba4b-271efe211e11-kube-api-access-5fcvn\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.621009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.621009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.620823    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-trusted-ca\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.721762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721669    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17b92591-8c5b-4fb5-ba4b-271efe211e11-config-volume\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.721762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721723    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcvn\" (UniqueName: \"kubernetes.io/projected/17b92591-8c5b-4fb5-ba4b-271efe211e11-kube-api-access-5fcvn\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.721762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721751    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.722052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721776    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-trusted-ca\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721826    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b05a9784-a678-478a-9694-00408b02e4cf-cert\") pod \"ingress-canary-fwtrx\" (UID: \"b05a9784-a678-478a-9694-00408b02e4cf\") " pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:39.722052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721849    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-certificates\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721877    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gsds\" (UniqueName: \"kubernetes.io/projected/b05a9784-a678-478a-9694-00408b02e4cf-kube-api-access-9gsds\") pod \"ingress-canary-fwtrx\" (UID: \"b05a9784-a678-478a-9694-00408b02e4cf\") " pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:39.722052 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.721906    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:39.722052 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.721985    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls podName:17b92591-8c5b-4fb5-ba4b-271efe211e11 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:40.221957574 +0000 UTC m=+37.694186341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls") pod "dns-default-lqt7x" (UID: "17b92591-8c5b-4fb5-ba4b-271efe211e11") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:39.722315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.721906    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36de539f-0164-4957-bb2d-e422fd60ea8e-ca-trust-extracted\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722130    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-bound-sa-token\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722203    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-image-registry-private-configuration\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722231    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722256    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5rw\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-kube-api-access-mf5rw\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722265    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36de539f-0164-4957-bb2d-e422fd60ea8e-ca-trust-extracted\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722315 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722301    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-installation-pull-secrets\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722603 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722332    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b92591-8c5b-4fb5-ba4b-271efe211e11-tmp-dir\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.722603 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722430    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17b92591-8c5b-4fb5-ba4b-271efe211e11-config-volume\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.722603 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.722484    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 20:19:39.722603 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.722498    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79bd95c96d-zwg5j: secret "image-registry-tls" not found
Jul 13 20:19:39.722603 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722529    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-certificates\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.722603 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:39.722538    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls podName:36de539f-0164-4957-bb2d-e422fd60ea8e nodeName:}" failed. No retries permitted until 2026-07-13 20:19:40.222524584 +0000 UTC m=+37.694753339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls") pod "image-registry-79bd95c96d-zwg5j" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e") : secret "image-registry-tls" not found
Jul 13 20:19:39.722603 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722598    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17b92591-8c5b-4fb5-ba4b-271efe211e11-tmp-dir\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.722959 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.722931    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-trusted-ca\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.727077 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.727051    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-installation-pull-secrets\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.727077 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.727067    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b05a9784-a678-478a-9694-00408b02e4cf-cert\") pod \"ingress-canary-fwtrx\" (UID: \"b05a9784-a678-478a-9694-00408b02e4cf\") " pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:39.727245 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.727056    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-image-registry-private-configuration\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.733856 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.733209    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcvn\" (UniqueName: \"kubernetes.io/projected/17b92591-8c5b-4fb5-ba4b-271efe211e11-kube-api-access-5fcvn\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:39.733856 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.733809    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gsds\" (UniqueName: \"kubernetes.io/projected/b05a9784-a678-478a-9694-00408b02e4cf-kube-api-access-9gsds\") pod \"ingress-canary-fwtrx\" (UID: \"b05a9784-a678-478a-9694-00408b02e4cf\") " pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:39.734370 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.734342    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5rw\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-kube-api-access-mf5rw\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.734645 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.734625    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-bound-sa-token\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:39.783430 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:39.783393    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fwtrx"
Jul 13 20:19:40.130957 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.130924    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zdd9x"]
Jul 13 20:19:40.134467 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.134442    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.136724 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.136701    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 20:19:40.137147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.137128    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 20:19:40.137414 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.137397    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4rrll\""
Jul 13 20:19:40.137490 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.137446    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 20:19:40.137687 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.137673    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 20:19:40.149368 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.149298    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdd9x"]
Jul 13 20:19:40.162410 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.162380    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:40.225998 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.225953    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.226167 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.226060    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z69sh\" (UniqueName: \"kubernetes.io/projected/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-api-access-z69sh\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.226246 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.226208    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:40.226310 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.226252    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/515fd3ec-14df-46c2-b4ff-5638b256d6a9-data-volume\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.226310 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.226304    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.226412 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.226332    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/515fd3ec-14df-46c2-b4ff-5638b256d6a9-crio-socket\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.226412 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.226366    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 20:19:40.226412 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.226381    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79bd95c96d-zwg5j: secret "image-registry-tls" not found
Jul 13 20:19:40.226412 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.226387    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:40.226605 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.226534    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:40.226605 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.226542    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls podName:36de539f-0164-4957-bb2d-e422fd60ea8e nodeName:}" failed. No retries permitted until 2026-07-13 20:19:41.226519175 +0000 UTC m=+38.698747930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls") pod "image-registry-79bd95c96d-zwg5j" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e") : secret "image-registry-tls" not found
Jul 13 20:19:40.226744 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.226617    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls podName:17b92591-8c5b-4fb5-ba4b-271efe211e11 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:41.226600295 +0000 UTC m=+38.698829050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls") pod "dns-default-lqt7x" (UID: "17b92591-8c5b-4fb5-ba4b-271efe211e11") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:40.236763 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.236725    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:40.327769 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.327743    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/515fd3ec-14df-46c2-b4ff-5638b256d6a9-data-volume\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.327870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.327788    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.327870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.327808    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/515fd3ec-14df-46c2-b4ff-5638b256d6a9-crio-socket\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.327870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.327842    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.327870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.327863    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z69sh\" (UniqueName: \"kubernetes.io/projected/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-api-access-z69sh\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.328071 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.328050    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:19:40.328162 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.328140    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:40.828118662 +0000 UTC m=+38.300347434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : configmap "kube-rbac-proxy" not found
Jul 13 20:19:40.328259 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.328243    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:40.328312 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.328291    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/515fd3ec-14df-46c2-b4ff-5638b256d6a9-crio-socket\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.328356 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.328306    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:40.828289453 +0000 UTC m=+38.300518221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:40.328433 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.328419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/515fd3ec-14df-46c2-b4ff-5638b256d6a9-data-volume\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.344953 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.344919    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z69sh\" (UniqueName: \"kubernetes.io/projected/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-api-access-z69sh\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.394293 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.394095    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fwtrx"]
Jul 13 20:19:40.397930 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:40.397858    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05a9784_a678_478a_9694_00408b02e4cf.slice/crio-cc2150aaa282c85a76fdeb3882437c1077a68c75b1379105f50f506e5cf70d67 WatchSource:0}: Error finding container cc2150aaa282c85a76fdeb3882437c1077a68c75b1379105f50f506e5cf70d67: Status 404 returned error can't find the container with id cc2150aaa282c85a76fdeb3882437c1077a68c75b1379105f50f506e5cf70d67
Jul 13 20:19:40.477076 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.477036    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:40.663102 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.663008    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:40.742324 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.742284    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:40.831694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.831640    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.831847 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:40.831710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:40.831847 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.831733    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:19:40.831847 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.831795    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:41.831779227 +0000 UTC m=+39.304007984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : configmap "kube-rbac-proxy" not found
Jul 13 20:19:40.831949 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.831861    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:40.831949 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:40.831918    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:41.831901738 +0000 UTC m=+39.304130499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:41.135944 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.135912    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:19:41.136599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.136019    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:19:41.136599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.136105    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:19:41.138872 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.138849    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhkt5\""
Jul 13 20:19:41.139007 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.138897    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 20:19:41.139726 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.139705    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 20:19:41.139835 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.139752    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 20:19:41.140437 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.140420    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwnrg\""
Jul 13 20:19:41.140543 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.140424    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 20:19:41.235129 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.235098    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:41.235298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.235215    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:41.235298 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.235259    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:41.235395 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.235324    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls podName:17b92591-8c5b-4fb5-ba4b-271efe211e11 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:43.235305021 +0000 UTC m=+40.707533796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls") pod "dns-default-lqt7x" (UID: "17b92591-8c5b-4fb5-ba4b-271efe211e11") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:41.235395 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.235326    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 20:19:41.235395 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.235339    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79bd95c96d-zwg5j: secret "image-registry-tls" not found
Jul 13 20:19:41.235395 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.235372    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls podName:36de539f-0164-4957-bb2d-e422fd60ea8e nodeName:}" failed. No retries permitted until 2026-07-13 20:19:43.235362083 +0000 UTC m=+40.707590849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls") pod "image-registry-79bd95c96d-zwg5j" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e") : secret "image-registry-tls" not found
Jul 13 20:19:41.287316 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.287272    2551 generic.go:356] "Generic (PLEG): container finished" podID="de209486-217b-4fb2-bb81-fb4f185312c1" containerID="d47ee4d296a588575b1b83bf416be7580c5ae283d5ac83393e4fedd47d0b2bd2" exitCode=0
Jul 13 20:19:41.287316 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.287315    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerDied","Data":"d47ee4d296a588575b1b83bf416be7580c5ae283d5ac83393e4fedd47d0b2bd2"}
Jul 13 20:19:41.288835 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.288806    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fwtrx" event={"ID":"b05a9784-a678-478a-9694-00408b02e4cf","Type":"ContainerStarted","Data":"cc2150aaa282c85a76fdeb3882437c1077a68c75b1379105f50f506e5cf70d67"}
Jul 13 20:19:41.409604 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.409523    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:41.473884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.473852    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:41.833114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.833027    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:41.841065 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.841031    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:41.841199 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.841097    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:41.841259 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.841236    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:41.841325 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.841296    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:43.841276922 +0000 UTC m=+41.313505694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:41.841691 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.841646    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:19:41.841784 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:41.841730    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:43.841714051 +0000 UTC m=+41.313942817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : configmap "kube-rbac-proxy" not found
Jul 13 20:19:41.936190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:41.936151    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:42.213035 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:42.212997    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:42.265111 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:42.265080    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-js24l_2066f3e0-e084-4802-8e22-bf7eec308a63/dns-node-resolver/0.log"
Jul 13 20:19:42.294891 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:42.294856    2551 generic.go:356] "Generic (PLEG): container finished" podID="de209486-217b-4fb2-bb81-fb4f185312c1" containerID="794978d11aaafdfc84f0613501d3f195fb6d287f38b6e666a27657245dd748da" exitCode=0
Jul 13 20:19:42.295043 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:42.294923    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerDied","Data":"794978d11aaafdfc84f0613501d3f195fb6d287f38b6e666a27657245dd748da"}
Jul 13 20:19:42.344383 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:42.344348    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:42.866038 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:42.866011    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5hd22_7c02c555-f585-4b0f-877d-766cecd345e7/node-ca/0.log"
Jul 13 20:19:43.143104 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.142911    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:43.254188 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.254100    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:43.254188 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.254176    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:43.254601 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.254243    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 20:19:43.254601 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.254260    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79bd95c96d-zwg5j: secret "image-registry-tls" not found
Jul 13 20:19:43.254601 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.254314    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls podName:36de539f-0164-4957-bb2d-e422fd60ea8e nodeName:}" failed. No retries permitted until 2026-07-13 20:19:47.254297926 +0000 UTC m=+44.726526682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls") pod "image-registry-79bd95c96d-zwg5j" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e") : secret "image-registry-tls" not found
Jul 13 20:19:43.254601 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.254321    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:43.254601 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.254404    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls podName:17b92591-8c5b-4fb5-ba4b-271efe211e11 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:47.254391148 +0000 UTC m=+44.726619901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls") pod "dns-default-lqt7x" (UID: "17b92591-8c5b-4fb5-ba4b-271efe211e11") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:43.298423 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.298394    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fwtrx" event={"ID":"b05a9784-a678-478a-9694-00408b02e4cf","Type":"ContainerStarted","Data":"1f1b26694ba618290c962df069256f2bb0acdaac01814b9a290dc0c78c0b20ce"}
Jul 13 20:19:43.301663 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.301624    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5crvv" event={"ID":"de209486-217b-4fb2-bb81-fb4f185312c1","Type":"ContainerStarted","Data":"541c38c99d23ff288db16e8528a2f35408aa3b61496a95734bfba93206c2ea6f"}
Jul 13 20:19:43.316452 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.316403    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fwtrx" podStartSLOduration=1.725896825 podStartE2EDuration="4.316387746s" totalImagesPullingTime="2.590490921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:43.315365218 +0000 UTC m=+40.787593993" watchObservedRunningTime="2026-07-13 20:19:43.316387746 +0000 UTC m=+40.788616520"
Jul 13 20:19:43.341210 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.341126    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5crvv" podStartSLOduration=9.443745354 podStartE2EDuration="40.341107076s" totalImagesPullingTime="30.897361722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:43.339583762 +0000 UTC m=+40.811812536" watchObservedRunningTime="2026-07-13 20:19:43.341107076 +0000 UTC m=+40.813335850"
Jul 13 20:19:43.470996 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.470962    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:43.568874 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.568798    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:43.665857 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.665825    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fwtrx_b05a9784-a678-478a-9694-00408b02e4cf/serve-healthcheck-canary/0.log"
Jul 13 20:19:43.711694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.711643    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:43.731456 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.731431    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-lkljj"]
Jul 13 20:19:43.753825 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.753799    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-lkljj"]
Jul 13 20:19:43.753825 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.753819    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.758994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.758969    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 13 20:19:43.759110 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.759092    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 13 20:19:43.759169 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.759152    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 13 20:19:43.759169 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.759161    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7bkfb\""
Jul 13 20:19:43.759276 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.759230    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 13 20:19:43.783602 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.783576    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:43.859628 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.859590    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:43.859762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.859633    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48r28\" (UniqueName: \"kubernetes.io/projected/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-kube-api-access-48r28\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.859762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.859687    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:43.859762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.859706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-signing-key\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.859762 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.859726    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 20:19:43.859762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.859752    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-signing-cabundle\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.859929 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.859788    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:47.859771981 +0000 UTC m=+45.332000766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : configmap "kube-rbac-proxy" not found
Jul 13 20:19:43.859929 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.859878    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:43.859994 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:43.859934    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:47.85992163 +0000 UTC m=+45.332150382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:43.961077 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.961041    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-signing-cabundle\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.961233 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.961155    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48r28\" (UniqueName: \"kubernetes.io/projected/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-kube-api-access-48r28\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.961233 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.961192    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-signing-key\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.961783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.961758    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-signing-cabundle\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.965221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.965199    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-signing-key\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:43.976379 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:43.976350    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48r28\" (UniqueName: \"kubernetes.io/projected/4f8619e4-0001-468b-ba74-e8a1a7c6dcc8-kube-api-access-48r28\") pod \"service-ca-79f68b9f4f-lkljj\" (UID: \"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8\") " pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:44.061844 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:44.061814    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj"
Jul 13 20:19:44.201901 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:44.201874    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-lkljj"]
Jul 13 20:19:44.205481 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:44.205457    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8619e4_0001_468b_ba74_e8a1a7c6dcc8.slice/crio-f98884751c1967b1cae749ac522e261a81094fde25129551afd7a2635b84466d WatchSource:0}: Error finding container f98884751c1967b1cae749ac522e261a81094fde25129551afd7a2635b84466d: Status 404 returned error can't find the container with id f98884751c1967b1cae749ac522e261a81094fde25129551afd7a2635b84466d
Jul 13 20:19:44.304696 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:44.304666    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj" event={"ID":"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8","Type":"ContainerStarted","Data":"f98884751c1967b1cae749ac522e261a81094fde25129551afd7a2635b84466d"}
Jul 13 20:19:45.074817 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:45.074768    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:45.583959 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:45.583922    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:45.776496 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:45.776460    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:45.848962 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:45.848929    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:47.288216 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:47.288178    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:47.288583 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:47.288243    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:47.288583 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:47.288330    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 13 20:19:47.288583 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:47.288349    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79bd95c96d-zwg5j: secret "image-registry-tls" not found
Jul 13 20:19:47.288583 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:47.288350    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 20:19:47.288583 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:47.288406    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls podName:17b92591-8c5b-4fb5-ba4b-271efe211e11 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:55.28838939 +0000 UTC m=+52.760618142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls") pod "dns-default-lqt7x" (UID: "17b92591-8c5b-4fb5-ba4b-271efe211e11") : secret "dns-default-metrics-tls" not found
Jul 13 20:19:47.288583 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:47.288425    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls podName:36de539f-0164-4957-bb2d-e422fd60ea8e nodeName:}" failed. No retries permitted until 2026-07-13 20:19:55.288416578 +0000 UTC m=+52.760645330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls") pod "image-registry-79bd95c96d-zwg5j" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e") : secret "image-registry-tls" not found
Jul 13 20:19:47.313597 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:47.313564    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj" event={"ID":"4f8619e4-0001-468b-ba74-e8a1a7c6dcc8","Type":"ContainerStarted","Data":"6e94cccb9cdc812766500ad91cfbe2ffc866e3aca33004e67460914b6453f24a"}
Jul 13 20:19:47.893391 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:47.893350    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:47.893597 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:47.893403    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:47.893597 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:47.893507    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:47.893597 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:19:47.893558    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls podName:515fd3ec-14df-46c2-b4ff-5638b256d6a9 nodeName:}" failed. No retries permitted until 2026-07-13 20:19:55.893544517 +0000 UTC m=+53.365773268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdd9x" (UID: "515fd3ec-14df-46c2-b4ff-5638b256d6a9") : secret "insights-runtime-extractor-tls" not found
Jul 13 20:19:47.893970 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:47.893947    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/515fd3ec-14df-46c2-b4ff-5638b256d6a9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:48.011715 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:48.011675    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:48.344355 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:48.344314    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:48.440015 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:48.439982    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:48.540312 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:48.540283    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:48.643177 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:48.643088    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:48.745957 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:48.745925    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:49.026720 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:49.026622    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:49.958812 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:49.958760    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:50.393882 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:50.393825    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:50.496979 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:50.496941    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:50.645612 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:50.645524    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:50.847594 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:50.847551    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:50.926139 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:50.926044    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:51.020193 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:51.020147    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:51.228870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:51.228778    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:51.329459 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:51.329418    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:51.630995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:51.630961    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:52.020910 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:52.020828    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:52.219697 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:52.219667    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:52.348776 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:52.348741    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:52.620325 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:52.620241    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:53.027229 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:53.027136    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:53.115226 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:53.115177    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:53.429765 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:53.429728    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:53.626386 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:53.626348    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:54.026769 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:54.026730    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:54.628473 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:54.628437    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:54.686614 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:54.686578    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:55.026970 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.026883    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:55.359676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.359628    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:55.359824 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.359731    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:55.362046 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.362011    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17b92591-8c5b-4fb5-ba4b-271efe211e11-metrics-tls\") pod \"dns-default-lqt7x\" (UID: \"17b92591-8c5b-4fb5-ba4b-271efe211e11\") " pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:55.362153 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.362061    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"image-registry-79bd95c96d-zwg5j\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") " pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:55.362933 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.362918    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:55.388797 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.388770    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:19:55.421384 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.421353    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:55.515099 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.515041    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-lkljj" podStartSLOduration=10.229953125 podStartE2EDuration="12.515020098s" totalImagesPullingTime="2.285066973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:47.337680286 +0000 UTC m=+44.809909058" watchObservedRunningTime="2026-07-13 20:19:55.515020098 +0000 UTC m=+52.987248873"
Jul 13 20:19:55.516112 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.516075    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79bd95c96d-zwg5j"]
Jul 13 20:19:55.519955 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:55.519927    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36de539f_0164_4957_bb2d_e422fd60ea8e.slice/crio-d7ce2eafba94d3039bc20a6c64c8276720400432f1795cf9989b9a10cc1cd2ea WatchSource:0}: Error finding container d7ce2eafba94d3039bc20a6c64c8276720400432f1795cf9989b9a10cc1cd2ea: Status 404 returned error can't find the container with id d7ce2eafba94d3039bc20a6c64c8276720400432f1795cf9989b9a10cc1cd2ea
Jul 13 20:19:55.558589 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.558566    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lqt7x"]
Jul 13 20:19:55.561812 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:55.561786    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b92591_8c5b_4fb5_ba4b_271efe211e11.slice/crio-752735ac4fc0ff80e9f6c22be16a46d832ee10e285b194e58cf69746d78e3831 WatchSource:0}: Error finding container 752735ac4fc0ff80e9f6c22be16a46d832ee10e285b194e58cf69746d78e3831: Status 404 returned error can't find the container with id 752735ac4fc0ff80e9f6c22be16a46d832ee10e285b194e58cf69746d78e3831
Jul 13 20:19:55.824698 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.824665    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:55.965460 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.965377    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:55.967631 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:55.967603    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/515fd3ec-14df-46c2-b4ff-5638b256d6a9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdd9x\" (UID: \"515fd3ec-14df-46c2-b4ff-5638b256d6a9\") " pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:56.022797 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.022753    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:56.045693 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.045646    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdd9x"
Jul 13 20:19:56.221630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.221543    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdd9x"]
Jul 13 20:19:56.245396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:19:56.245370    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515fd3ec_14df_46c2_b4ff_5638b256d6a9.slice/crio-e85c3d1429a81e85e026e60b35a03fe2f590a9dd67cc7bf44bd3482c7706b341 WatchSource:0}: Error finding container e85c3d1429a81e85e026e60b35a03fe2f590a9dd67cc7bf44bd3482c7706b341: Status 404 returned error can't find the container with id e85c3d1429a81e85e026e60b35a03fe2f590a9dd67cc7bf44bd3482c7706b341
Jul 13 20:19:56.342919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.342873    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdd9x" event={"ID":"515fd3ec-14df-46c2-b4ff-5638b256d6a9","Type":"ContainerStarted","Data":"e85c3d1429a81e85e026e60b35a03fe2f590a9dd67cc7bf44bd3482c7706b341"}
Jul 13 20:19:56.344157 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.344137    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqt7x" event={"ID":"17b92591-8c5b-4fb5-ba4b-271efe211e11","Type":"ContainerStarted","Data":"752735ac4fc0ff80e9f6c22be16a46d832ee10e285b194e58cf69746d78e3831"}
Jul 13 20:19:56.345627 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.345602    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j" event={"ID":"36de539f-0164-4957-bb2d-e422fd60ea8e","Type":"ContainerStarted","Data":"6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25"}
Jul 13 20:19:56.345716 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.345641    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j" event={"ID":"36de539f-0164-4957-bb2d-e422fd60ea8e","Type":"ContainerStarted","Data":"d7ce2eafba94d3039bc20a6c64c8276720400432f1795cf9989b9a10cc1cd2ea"}
Jul 13 20:19:56.345762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.345735    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:19:56.375761 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.375710    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j" podStartSLOduration=39.37558696 podStartE2EDuration="39.37558696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:19:56.375243265 +0000 UTC m=+53.847472039" watchObservedRunningTime="2026-07-13 20:19:56.37558696 +0000 UTC m=+53.847815726"
Jul 13 20:19:56.433448 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:56.433408    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:57.051732 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:57.051694    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:57.432536 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:57.432454    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:57.623647 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:57.623526    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:58.231260 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:58.231228    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:58.623003 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:58.622966    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:59.325390 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:59.325354    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:19:59.432921 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:19:59.432876    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:01.364135 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:01.364103    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdd9x" event={"ID":"515fd3ec-14df-46c2-b4ff-5638b256d6a9","Type":"ContainerStarted","Data":"d17dfb3513ec791a26aa0d3e667f79de49f3182a575ee1b90a4cf91baab0855a"}
Jul 13 20:20:01.365831 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:01.365799    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqt7x" event={"ID":"17b92591-8c5b-4fb5-ba4b-271efe211e11","Type":"ContainerStarted","Data":"79a30e4e6c8ba18e32043ac4fc239cccf2305726247cdaacc9effb39fa2bb4d2"}
Jul 13 20:20:01.365947 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:01.365853    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lqt7x" event={"ID":"17b92591-8c5b-4fb5-ba4b-271efe211e11","Type":"ContainerStarted","Data":"2b279e2cc65a107da5744c610132195213ddcd16ffdc38c208ea453f2105f961"}
Jul 13 20:20:01.365947 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:01.365909    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:20:01.398002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:01.397956    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lqt7x" podStartSLOduration=17.507299749 podStartE2EDuration="22.397941479s" totalImagesPullingTime="4.89064173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:01.397556593 +0000 UTC m=+58.869785367" watchObservedRunningTime="2026-07-13 20:20:01.397941479 +0000 UTC m=+58.870170274"
Jul 13 20:20:02.370485 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:02.370438    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdd9x" event={"ID":"515fd3ec-14df-46c2-b4ff-5638b256d6a9","Type":"ContainerStarted","Data":"e57ae1d3302e8d6c5b58b66a39472a578d41390264d431b1711250f42485efcc"}
Jul 13 20:20:03.283835 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.283805    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zmz6m"
Jul 13 20:20:03.364788 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.364751    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"]
Jul 13 20:20:03.366974 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.366953    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"]
Jul 13 20:20:03.367166 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.367078    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.368915 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.368892    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.370694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.370671    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 13 20:20:03.371060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.370841    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 13 20:20:03.372098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.372076    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 13 20:20:03.373121 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.373100    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 13 20:20:03.373345 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.373324    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 13 20:20:03.374392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.374344    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 13 20:20:03.374392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.374379    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 13 20:20:03.375224 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.375202    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 13 20:20:03.379428 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.379412    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79bd95c96d-zwg5j"]
Jul 13 20:20:03.384268 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.384202    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"]
Jul 13 20:20:03.394049 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.394024    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"]
Jul 13 20:20:03.427828 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.427799    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.428025 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.427838    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20de037d-e73d-4efe-902b-f218704c4790-klusterlet-config\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.428025 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.427939    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-hub\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.428025 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.428003    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44e48e4-157f-443b-abf3-4067182da683-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.428167 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.428044    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpd8j\" (UniqueName: \"kubernetes.io/projected/c44e48e4-157f-443b-abf3-4067182da683-kube-api-access-qpd8j\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.428167 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.428062    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-ca\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.428167 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.428086    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20de037d-e73d-4efe-902b-f218704c4790-tmp\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.428167 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.428142    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.428366 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.428267    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v8qq\" (UniqueName: \"kubernetes.io/projected/20de037d-e73d-4efe-902b-f218704c4790-kube-api-access-8v8qq\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.529646 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.529613    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44e48e4-157f-443b-abf3-4067182da683-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.529866 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.529685    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpd8j\" (UniqueName: \"kubernetes.io/projected/c44e48e4-157f-443b-abf3-4067182da683-kube-api-access-qpd8j\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.529866 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.529714    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-ca\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.529866 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.529743    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20de037d-e73d-4efe-902b-f218704c4790-tmp\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.529866 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.529775    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.530068 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.530028    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v8qq\" (UniqueName: \"kubernetes.io/projected/20de037d-e73d-4efe-902b-f218704c4790-kube-api-access-8v8qq\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.530125 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.530067    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.530125 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.530098    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20de037d-e73d-4efe-902b-f218704c4790-klusterlet-config\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.530232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.530150    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-hub\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.530232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.530179    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20de037d-e73d-4efe-902b-f218704c4790-tmp\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.530705 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.530429    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44e48e4-157f-443b-abf3-4067182da683-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.532914 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.532749    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.533070 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.533047    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-ca\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.533150 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.533108    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.533255 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.533176    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c44e48e4-157f-443b-abf3-4067182da683-hub\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.533327 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.533304    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/20de037d-e73d-4efe-902b-f218704c4790-klusterlet-config\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.543190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.543118    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpd8j\" (UniqueName: \"kubernetes.io/projected/c44e48e4-157f-443b-abf3-4067182da683-kube-api-access-qpd8j\") pod \"cluster-proxy-proxy-agent-7675bcfd8f-vd7h8\" (UID: \"c44e48e4-157f-443b-abf3-4067182da683\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.552725 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.552697    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v8qq\" (UniqueName: \"kubernetes.io/projected/20de037d-e73d-4efe-902b-f218704c4790-kube-api-access-8v8qq\") pod \"klusterlet-addon-workmgr-7455df9ddc-w5fbz\" (UID: \"20de037d-e73d-4efe-902b-f218704c4790\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.571553 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.571520    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:03.680021 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.679985    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:03.692824 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.692789    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"
Jul 13 20:20:03.697982 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.697945    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:03.800508 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.800333    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:03.839858 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.839827    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"]
Jul 13 20:20:03.843130 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:20:03.843100    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20de037d_e73d_4efe_902b_f218704c4790.slice/crio-da3fde178de12101af1832c164bd92ed99d5b418d3915b6ed49acc4c07e6429d WatchSource:0}: Error finding container da3fde178de12101af1832c164bd92ed99d5b418d3915b6ed49acc4c07e6429d: Status 404 returned error can't find the container with id da3fde178de12101af1832c164bd92ed99d5b418d3915b6ed49acc4c07e6429d
Jul 13 20:20:03.879217 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.879186    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8"]
Jul 13 20:20:03.883784 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:03.883751    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:03.883917 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:20:03.883821    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc44e48e4_157f_443b_abf3_4067182da683.slice/crio-c6f467ee8880b3d56fadeb930f7cdd44cb00afe9090c8a412158acbe8a5680ba WatchSource:0}: Error finding container c6f467ee8880b3d56fadeb930f7cdd44cb00afe9090c8a412158acbe8a5680ba: Status 404 returned error can't find the container with id c6f467ee8880b3d56fadeb930f7cdd44cb00afe9090c8a412158acbe8a5680ba
Jul 13 20:20:04.378950 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:04.378912    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz" event={"ID":"20de037d-e73d-4efe-902b-f218704c4790","Type":"ContainerStarted","Data":"da3fde178de12101af1832c164bd92ed99d5b418d3915b6ed49acc4c07e6429d"}
Jul 13 20:20:04.380188 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:04.380163    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8" event={"ID":"c44e48e4-157f-443b-abf3-4067182da683","Type":"ContainerStarted","Data":"c6f467ee8880b3d56fadeb930f7cdd44cb00afe9090c8a412158acbe8a5680ba"}
Jul 13 20:20:04.568617 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:04.568577    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:04.642443 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:04.642349    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:04.819986 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:04.819941    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:04.917913 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:04.917811    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:05.383873 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:05.383839    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:05.487292 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:05.487242    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:06.390744 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:06.389544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdd9x" event={"ID":"515fd3ec-14df-46c2-b4ff-5638b256d6a9","Type":"ContainerStarted","Data":"df4bba15bd3a671e77413eb61222739c14af2723971589721668feffaa3b2fff"}
Jul 13 20:20:07.941670 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:07.941615    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:08.009982 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.009945    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:08.093167 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.093114    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:08.184532 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.184490    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:08.344667 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.344622    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:08.781209 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.781124    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:20:08.781367 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.781221    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:20:08.784008 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.783982    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 20:20:08.784340 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.784315    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 20:20:08.794488 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.794464    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 20:20:08.794980 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.794957    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc4b6b4f-5b06-4ace-8cef-2964b68d10fb-metrics-certs\") pod \"network-metrics-daemon-6wlbd\" (UID: \"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb\") " pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:20:08.805741 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.805707    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwd8w\" (UniqueName: \"kubernetes.io/projected/2771d026-f54c-4c67-a476-d3f5742dab82-kube-api-access-vwd8w\") pod \"network-check-target-xqc68\" (UID: \"2771d026-f54c-4c67-a476-d3f5742dab82\") " pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:20:08.881853 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.881819    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:20:08.884761 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.884740    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 20:20:08.895337 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:08.895308    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bf83bc3-42e1-409e-a511-bce2009ba373-original-pull-secret\") pod \"global-pull-secret-syncer-vjwb9\" (UID: \"5bf83bc3-42e1-409e-a511-bce2009ba373\") " pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:20:09.049915 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:09.049840    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhkt5\""
Jul 13 20:20:09.056709 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:09.056681    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwnrg\""
Jul 13 20:20:09.057610 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:09.057589    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:20:09.060691 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:09.060670    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjwb9"
Jul 13 20:20:09.065284 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:09.065266    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wlbd"
Jul 13 20:20:09.314082 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:09.314003    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:09.370447 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:09.370397    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:10.133293 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.133248    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zdd9x" podStartSLOduration=20.934053539 podStartE2EDuration="30.133232015s" totalImagesPullingTime="9.199178476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:06.415350448 +0000 UTC m=+63.887579223" watchObservedRunningTime="2026-07-13 20:20:10.133232015 +0000 UTC m=+67.605460789"
Jul 13 20:20:10.134235 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.134088    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xqc68"]
Jul 13 20:20:10.138431 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:20:10.138395    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2771d026_f54c_4c67_a476_d3f5742dab82.slice/crio-0bfebbbe2965727eb6d7a6dfd895dcd8544f979c80a835c440ad554d39c8ba53 WatchSource:0}: Error finding container 0bfebbbe2965727eb6d7a6dfd895dcd8544f979c80a835c440ad554d39c8ba53: Status 404 returned error can't find the container with id 0bfebbbe2965727eb6d7a6dfd895dcd8544f979c80a835c440ad554d39c8ba53
Jul 13 20:20:10.153094 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.153057    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjwb9"]
Jul 13 20:20:10.154324 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.154229    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wlbd"]
Jul 13 20:20:10.156169 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:20:10.156142    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4b6b4f_5b06_4ace_8cef_2964b68d10fb.slice/crio-62c4c37f979552b7343feaa7edb8d28995be9a1981117ad6f17583aeee89e0e1 WatchSource:0}: Error finding container 62c4c37f979552b7343feaa7edb8d28995be9a1981117ad6f17583aeee89e0e1: Status 404 returned error can't find the container with id 62c4c37f979552b7343feaa7edb8d28995be9a1981117ad6f17583aeee89e0e1
Jul 13 20:20:10.159618 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:20:10.159584    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf83bc3_42e1_409e_a511_bce2009ba373.slice/crio-f9602e49a6d3f44f3d6bc6a4010a273d1639697e4198841d66c940d9d2675152 WatchSource:0}: Error finding container f9602e49a6d3f44f3d6bc6a4010a273d1639697e4198841d66c940d9d2675152: Status 404 returned error can't find the container with id f9602e49a6d3f44f3d6bc6a4010a273d1639697e4198841d66c940d9d2675152
Jul 13 20:20:10.406288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.406246    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8" event={"ID":"c44e48e4-157f-443b-abf3-4067182da683","Type":"ContainerStarted","Data":"175c67549c32eee22eaa0566c267895c2b052f7153a9fe7daf10d933d699db28"}
Jul 13 20:20:10.406288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.406286    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8" event={"ID":"c44e48e4-157f-443b-abf3-4067182da683","Type":"ContainerStarted","Data":"b285408a8bfb2348b7376fdd8d8b5f6f9a0123bbdb9a310f4840045d3272c644"}
Jul 13 20:20:10.406288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.406299    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8" event={"ID":"c44e48e4-157f-443b-abf3-4067182da683","Type":"ContainerStarted","Data":"9663ee974bf93cc954b59dbc72c337df56d19d2f9a2fb50529fcc0afec7d89cf"}
Jul 13 20:20:10.407333 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.407306    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjwb9" event={"ID":"5bf83bc3-42e1-409e-a511-bce2009ba373","Type":"ContainerStarted","Data":"f9602e49a6d3f44f3d6bc6a4010a273d1639697e4198841d66c940d9d2675152"}
Jul 13 20:20:10.408338 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.408315    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wlbd" event={"ID":"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb","Type":"ContainerStarted","Data":"62c4c37f979552b7343feaa7edb8d28995be9a1981117ad6f17583aeee89e0e1"}
Jul 13 20:20:10.409371 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.409351    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqc68" event={"ID":"2771d026-f54c-4c67-a476-d3f5742dab82","Type":"ContainerStarted","Data":"0bfebbbe2965727eb6d7a6dfd895dcd8544f979c80a835c440ad554d39c8ba53"}
Jul 13 20:20:10.410594 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.410577    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz" event={"ID":"20de037d-e73d-4efe-902b-f218704c4790","Type":"ContainerStarted","Data":"dd1899e0a972dff7f7c348f0136c84468b93552f4f48145f61909610cb909fac"}
Jul 13 20:20:10.410813 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.410788    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:10.412412 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.412397    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz"
Jul 13 20:20:10.428455 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.428418    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7675bcfd8f-vd7h8" podStartSLOduration=1.350744438 podStartE2EDuration="7.428406699s" totalImagesPullingTime="6.077662261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:10.427327104 +0000 UTC m=+67.899555876" watchObservedRunningTime="2026-07-13 20:20:10.428406699 +0000 UTC m=+67.900635464"
Jul 13 20:20:10.443994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:10.443956    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7455df9ddc-w5fbz" podStartSLOduration=1.335653087 podStartE2EDuration="7.443943889s" totalImagesPullingTime="6.108290802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:10.44332266 +0000 UTC m=+67.915551428" watchObservedRunningTime="2026-07-13 20:20:10.443943889 +0000 UTC m=+67.916172662"
Jul 13 20:20:11.185782 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:11.185681    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:11.295766 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:11.295673    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:11.373632 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:11.373354    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lqt7x"
Jul 13 20:20:11.465799 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:11.465748    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:12.420961 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:12.420921    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wlbd" event={"ID":"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb","Type":"ContainerStarted","Data":"8d79d0cd6fffd8edc1ff22fa32717f01ffb65c1986ad967a9bd91d931418b4a2"}
Jul 13 20:20:12.421358 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:12.420982    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wlbd" event={"ID":"bc4b6b4f-5b06-4ace-8cef-2964b68d10fb","Type":"ContainerStarted","Data":"3b45c63338d5b5e1cdfcdde2f80872e86339bb3c4c82c7112bd337d6c50330d8"}
Jul 13 20:20:12.445296 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:12.445239    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6wlbd" podStartSLOduration=68.281202433 podStartE2EDuration="1m9.445217292s" totalImagesPullingTime="1.164014859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:12.44493204 +0000 UTC m=+69.917160816" watchObservedRunningTime="2026-07-13 20:20:12.445217292 +0000 UTC m=+69.917446068"
Jul 13 20:20:12.616092 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:12.616054    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:12.747181 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:12.747087    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:12.839563 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:12.839521    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:12.927396 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:12.927354    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:13.142103 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.142061    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:13.213779 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.213742    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:13.386422 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.386386    2551 patch_prober.go:28] interesting pod/image-registry-79bd95c96d-zwg5j container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 13 20:20:13.386600 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.386452    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j" podUID="36de539f-0164-4957-bb2d-e422fd60ea8e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:20:13.503514 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.503430    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:13.577935 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.577895    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:13.714467 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.714432    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:13.893666 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:13.893623    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:14.000183 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:14.000141    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:14.102556 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:14.102514    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:14.172973 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:14.172882    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:14.331344 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:14.331307    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:14.431875 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:14.431777    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:15.109123 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.109088    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:15.307409 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.307325    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:15.407845 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.407818    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:15.432778 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.432743    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjwb9" event={"ID":"5bf83bc3-42e1-409e-a511-bce2009ba373","Type":"ContainerStarted","Data":"854b875599a5e6eb51525155f816b44ad343f711846bbe876dd889a112c154ad"}
Jul 13 20:20:15.434266 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.434239    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xqc68" event={"ID":"2771d026-f54c-4c67-a476-d3f5742dab82","Type":"ContainerStarted","Data":"b2d0bfafa0ceca20d7f4ed7b48a7602d83df3e53adf2a032f0e1e7ad6a206221"}
Jul 13 20:20:15.434369 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.434299    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:20:15.453282 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.453240    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vjwb9" podStartSLOduration=66.717693869 podStartE2EDuration="1m11.453226247s" totalImagesPullingTime="4.735532378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:15.451606841 +0000 UTC m=+72.923835615" watchObservedRunningTime="2026-07-13 20:20:15.453226247 +0000 UTC m=+72.925455020"
Jul 13 20:20:15.467198 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.467152    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xqc68" podStartSLOduration=67.713889368 podStartE2EDuration="1m12.467140124s" totalImagesPullingTime="4.753250756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:15.466533061 +0000 UTC m=+72.938761834" watchObservedRunningTime="2026-07-13 20:20:15.467140124 +0000 UTC m=+72.939368901"
Jul 13 20:20:15.693298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:15.693247    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:16.529092 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:16.529054    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:16.600023 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:16.599984    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:17.521431 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.521395    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:17.642317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.642277    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:17.803487 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.803403    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:17.903995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.903960    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mgfwr"]
Jul 13 20:20:17.906520 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.906500    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.908075 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.908029    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:17.917298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.917105    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9nxr9\""
Jul 13 20:20:17.917298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.917205    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 20:20:17.917298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.917263    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 20:20:17.917560 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.917546    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 20:20:17.917636 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.917617    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 20:20:17.918323 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.918301    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 20:20:17.920519 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.920487    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 20:20:17.956923 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.956888    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.956923 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.956924    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-tls\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.957178 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.956964    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-metrics-client-ca\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.957178 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.957057    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.957178 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.957094    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-textfile\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.957178 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.957119    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-wtmp\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.957356 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.957182    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-root\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.957356 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.957206    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccgj\" (UniqueName: \"kubernetes.io/projected/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-kube-api-access-hccgj\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.957356 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.957238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-sys\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:17.999341 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:17.999309    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:18.057750 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057663    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-root\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.057750 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057702    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccgj\" (UniqueName: \"kubernetes.io/projected/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-kube-api-access-hccgj\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.057750 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057724    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-sys\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.057750 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057749    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057777    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-tls\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057810    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-root\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057816    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-metrics-client-ca\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057831    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-sys\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057906    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:20:18.057930    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057941    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-textfile\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.057966    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-wtmp\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058100 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:20:18.058006    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-tls podName:1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7 nodeName:}" failed. No retries permitted until 2026-07-13 20:20:18.557983028 +0000 UTC m=+76.030211793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-tls") pod "node-exporter-mgfwr" (UID: "1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7") : secret "node-exporter-tls" not found
Jul 13 20:20:18.058537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.058335    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-textfile\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.058336    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-wtmp\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.058482    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-metrics-client-ca\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.058537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.058505    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.060639 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.060615    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.073163 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.073142    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccgj\" (UniqueName: \"kubernetes.io/projected/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-kube-api-access-hccgj\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.094250 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.094222    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:18.245995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.245962    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:18.347076 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.347042    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:18.471078 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.471047    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:18.563009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.562968    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-tls\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.565284 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.565255    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7-node-exporter-tls\") pod \"node-exporter-mgfwr\" (UID: \"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7\") " pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.791150 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.791070    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:18.816666 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.816630    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgfwr"
Jul 13 20:20:18.827042 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:20:18.827013    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3fdf6a_ab13_43f6_b2a1_73b02e72a5f7.slice/crio-6abd43bf5eebc21e4edd60641db67947474b4c3f1dcc6b79c47ac43325533a8c WatchSource:0}: Error finding container 6abd43bf5eebc21e4edd60641db67947474b4c3f1dcc6b79c47ac43325533a8c: Status 404 returned error can't find the container with id 6abd43bf5eebc21e4edd60641db67947474b4c3f1dcc6b79c47ac43325533a8c
Jul 13 20:20:18.900551 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.900513    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:18.992446 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:18.992407    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:19.105083 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:19.105046    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:19.449936 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:19.449848    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgfwr" event={"ID":"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7","Type":"ContainerStarted","Data":"6abd43bf5eebc21e4edd60641db67947474b4c3f1dcc6b79c47ac43325533a8c"}
Jul 13 20:20:20.455002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:20.454968    2551 generic.go:356] "Generic (PLEG): container finished" podID="1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7" containerID="642b9263393fdf2e698b8002bdb9e4b07941cb5b66d0af132783367c93bca03d" exitCode=0
Jul 13 20:20:20.455442 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:20.455063    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgfwr" event={"ID":"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7","Type":"ContainerDied","Data":"642b9263393fdf2e698b8002bdb9e4b07941cb5b66d0af132783367c93bca03d"}
Jul 13 20:20:20.844079 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:20.844044    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:20.945828 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:20.945796    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:21.089096 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:21.087215    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:21.189932 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:21.189845    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:21.307507 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:21.307477    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:21.399036 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:21.399008    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:21.460616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:21.460540    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgfwr" event={"ID":"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7","Type":"ContainerStarted","Data":"702c897b0020c4945f42953b9681acfeab8da728207ce74f9cb8e6ee84b593f8"}
Jul 13 20:20:21.460616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:21.460604    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgfwr" event={"ID":"1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7","Type":"ContainerStarted","Data":"1189b0b982c1e74462ff443ce12bca6b03978344b71529d46011a12730b003ef"}
Jul 13 20:20:21.485145 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:21.485095    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mgfwr" podStartSLOduration=3.660516175 podStartE2EDuration="4.485078785s" totalImagesPullingTime="761.086931ms" totalInitContainerRuntime="63.475679ms" isStatefulPod=true podCreationTimestamp="2026-07-13 20:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:20:21.48347215 +0000 UTC m=+78.955700923" watchObservedRunningTime="2026-07-13 20:20:21.485078785 +0000 UTC m=+78.957307558"
Jul 13 20:20:22.569189 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:22.569150    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:22.672239 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:22.672207    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:22.931691 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:22.931642    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:23.383725 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:23.383700    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:20:24.512002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:24.511880    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:25.313884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:25.313841    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:25.464983 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:25.464946    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:25.522662 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:25.522621    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:26.683439 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:26.683408    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:26.753203 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:26.753170    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:27.358323 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:27.358283    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:27.420389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:27.420361    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:28.405153 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.405112    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j" podUID="36de539f-0164-4957-bb2d-e422fd60ea8e" containerName="registry" containerID="cri-o://6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25" gracePeriod=30
Jul 13 20:20:28.642573 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.642549    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:20:28.740779 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.740689    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-trusted-ca\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.740779 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.740753    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-image-registry-private-configuration\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.740779 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.740780    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf5rw\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-kube-api-access-mf5rw\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.741043 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.740797    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.741043 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.740826    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-installation-pull-secrets\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.741043 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.740863    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36de539f-0164-4957-bb2d-e422fd60ea8e-ca-trust-extracted\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.741043 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.741022    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-certificates\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.741232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.741063    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-bound-sa-token\") pod \"36de539f-0164-4957-bb2d-e422fd60ea8e\" (UID: \"36de539f-0164-4957-bb2d-e422fd60ea8e\") "
Jul 13 20:20:28.741232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.741185    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:20:28.741347 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.741318    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-trusted-ca\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:28.741590 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.741556    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:20:28.743440 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.743407    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:20:28.743551 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.743452    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-kube-api-access-mf5rw" (OuterVolumeSpecName: "kube-api-access-mf5rw") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "kube-api-access-mf5rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:20:28.743551 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.743519    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:20:28.743709 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.743689    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:20:28.743763 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.743729    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:20:28.752672 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.752627    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36de539f-0164-4957-bb2d-e422fd60ea8e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "36de539f-0164-4957-bb2d-e422fd60ea8e" (UID: "36de539f-0164-4957-bb2d-e422fd60ea8e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:20:28.841884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.841838    2551 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-image-registry-private-configuration\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:28.841884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.841880    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf5rw\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-kube-api-access-mf5rw\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:28.841884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.841891    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:28.841884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.841900    2551 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36de539f-0164-4957-bb2d-e422fd60ea8e-installation-pull-secrets\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:28.842101 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.841909    2551 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36de539f-0164-4957-bb2d-e422fd60ea8e-ca-trust-extracted\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:28.842101 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.841917    2551 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36de539f-0164-4957-bb2d-e422fd60ea8e-registry-certificates\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:28.842101 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:28.841925    2551 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36de539f-0164-4957-bb2d-e422fd60ea8e-bound-sa-token\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:20:29.492493 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.492403    2551 generic.go:356] "Generic (PLEG): container finished" podID="36de539f-0164-4957-bb2d-e422fd60ea8e" containerID="6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25" exitCode=0
Jul 13 20:20:29.492493 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.492469    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j"
Jul 13 20:20:29.492976 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.492490    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j" event={"ID":"36de539f-0164-4957-bb2d-e422fd60ea8e","Type":"ContainerDied","Data":"6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25"}
Jul 13 20:20:29.492976 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.492544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79bd95c96d-zwg5j" event={"ID":"36de539f-0164-4957-bb2d-e422fd60ea8e","Type":"ContainerDied","Data":"d7ce2eafba94d3039bc20a6c64c8276720400432f1795cf9989b9a10cc1cd2ea"}
Jul 13 20:20:29.492976 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.492565    2551 scope.go:117] "RemoveContainer" containerID="6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25"
Jul 13 20:20:29.500232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.500202    2551 scope.go:117] "RemoveContainer" containerID="6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25"
Jul 13 20:20:29.500490 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:20:29.500471    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25\": container with ID starting with 6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25 not found: ID does not exist" containerID="6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25"
Jul 13 20:20:29.500555 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.500497    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25"} err="failed to get container status \"6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25\": rpc error: code = NotFound desc = could not find container \"6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25\": container with ID starting with 6012d09fc8c2adb3ca7cd197f363f860116d90d6b94ad9a96845727959880d25 not found: ID does not exist"
Jul 13 20:20:29.540509 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.540482    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79bd95c96d-zwg5j"]
Jul 13 20:20:29.545521 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.545495    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79bd95c96d-zwg5j"]
Jul 13 20:20:29.591150 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:29.591121    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:30.082612 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:30.082582    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:30.179886 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:30.179852    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:30.317493 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:30.317443    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:30.541140 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:30.541112    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:30.863565 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:30.863532    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:30.928858 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:30.928819    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:31.135885 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:31.135799    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36de539f-0164-4957-bb2d-e422fd60ea8e" path="/var/lib/kubelet/pods/36de539f-0164-4957-bb2d-e422fd60ea8e/volumes"
Jul 13 20:20:33.174743 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:33.174710    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:33.275438 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:33.275396    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:35.405952 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:35.405921    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:35.505886 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:35.505858    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:35.604940 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:35.604902    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:39.809267 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:39.809236    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:39.905849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:39.905814    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:42.645116 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:42.645083    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:42.721064 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:42.721030    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:46.440643 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:46.440610    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xqc68"
Jul 13 20:20:49.738206 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:49.738151    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:49.839028 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:49.838990    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:50.392599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:50.392539    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:50.520285 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:50.520197    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:50.749537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:50.749452    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:50.822691 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:50.822643    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:51.527885 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:51.527849    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:20:51.585962 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:20:51.585932    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:33.331910 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:21:33.331877    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:34.287131 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:21:34.287099    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:34.349628 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:21:34.349570    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:46.241866 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:21:46.241820    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:46.334783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:21:46.334726    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:50.385909 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:21:50.385871    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:21:50.487205 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:21:50.487177    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:13.520975 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:22:13.520860    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:13.592457 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:22:13.592425    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:24.724942 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:22:24.724901    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:24.857875 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:22:24.857837    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:31.201121 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:22:31.201078    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:22:31.301978 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:22:31.301938    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:04.353909 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:04.353877    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:12.008289 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:12.008234    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:12.145569 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:12.145536    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:19.064707 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:19.064669    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:33.421185 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:33.421134    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:33.522046 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:33.522005    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:33.896846 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:33.896812    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:33.998431 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:33.998404    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:23:34.090441 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:23:34.090407    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:01.775133 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:01.775088    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:02.928728 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:02.928701    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:24:02.931928 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:02.931904    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:24:02.935044 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:02.935026    2551 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 20:24:04.212068 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:04.212034    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:07.954606 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:07.954564    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:08.290044 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:08.289956    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:08.502180 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:08.502135    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:08.761013 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:08.760979    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:08.848729 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:08.848696    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:09.175530 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:09.175495    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:09.267328 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:09.267286    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:09.527910 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:09.527821    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:09.619909 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:09.619873    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:09.727831 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:09.727800    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:09.916598 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:09.916557    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:10.013159 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:10.013127    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:24:10.110887 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:24:10.110853    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:19.089208 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:19.089125    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:21.232537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.232500    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:21.333334 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.333301    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:21.514621 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.514537    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:21.600170 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.600139    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:21.617180 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.617156    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-vwxlw"]
Jul 13 20:25:21.617479 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.617465    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36de539f-0164-4957-bb2d-e422fd60ea8e" containerName="registry"
Jul 13 20:25:21.617541 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.617481    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="36de539f-0164-4957-bb2d-e422fd60ea8e" containerName="registry"
Jul 13 20:25:21.617541 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.617539    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="36de539f-0164-4957-bb2d-e422fd60ea8e" containerName="registry"
Jul 13 20:25:21.619445 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.619423    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:21.626001 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.625827    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 13 20:25:21.626395 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.626375    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-phc2s\""
Jul 13 20:25:21.626507 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.626399    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 13 20:25:21.626636 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.626620    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 13 20:25:21.643230 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.643205    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbb99aa6-4589-4999-bd22-cd10fc524542-cert\") pod \"llmisvc-controller-manager-84fc959947-vwxlw\" (UID: \"cbb99aa6-4589-4999-bd22-cd10fc524542\") " pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:21.643335 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.643238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sj8r\" (UniqueName: \"kubernetes.io/projected/cbb99aa6-4589-4999-bd22-cd10fc524542-kube-api-access-4sj8r\") pod \"llmisvc-controller-manager-84fc959947-vwxlw\" (UID: \"cbb99aa6-4589-4999-bd22-cd10fc524542\") " pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:21.672054 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.672005    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-vwxlw"]
Jul 13 20:25:21.673161 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.673144    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-tnckk"]
Jul 13 20:25:21.675225 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.675206    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:21.678661 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.678630    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 13 20:25:21.681451 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.681430    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cg5sz\""
Jul 13 20:25:21.690227 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.690206    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tnckk"]
Jul 13 20:25:21.730224 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.730198    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:21.743926 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.743907    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vch\" (UniqueName: \"kubernetes.io/projected/4d64c846-a288-44f3-8857-a483217d8080-kube-api-access-n4vch\") pod \"seaweedfs-64568bcd49-tnckk\" (UID: \"4d64c846-a288-44f3-8857-a483217d8080\") " pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:21.744040 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.743941    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sj8r\" (UniqueName: \"kubernetes.io/projected/cbb99aa6-4589-4999-bd22-cd10fc524542-kube-api-access-4sj8r\") pod \"llmisvc-controller-manager-84fc959947-vwxlw\" (UID: \"cbb99aa6-4589-4999-bd22-cd10fc524542\") " pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:21.744040 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.743979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d64c846-a288-44f3-8857-a483217d8080-data\") pod \"seaweedfs-64568bcd49-tnckk\" (UID: \"4d64c846-a288-44f3-8857-a483217d8080\") " pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:21.744147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.744123    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbb99aa6-4589-4999-bd22-cd10fc524542-cert\") pod \"llmisvc-controller-manager-84fc959947-vwxlw\" (UID: \"cbb99aa6-4589-4999-bd22-cd10fc524542\") " pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:21.744245 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:25:21.744231    2551 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 13 20:25:21.744313 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:25:21.744302    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbb99aa6-4589-4999-bd22-cd10fc524542-cert podName:cbb99aa6-4589-4999-bd22-cd10fc524542 nodeName:}" failed. No retries permitted until 2026-07-13 20:25:22.244281866 +0000 UTC m=+379.716510624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cbb99aa6-4589-4999-bd22-cd10fc524542-cert") pod "llmisvc-controller-manager-84fc959947-vwxlw" (UID: "cbb99aa6-4589-4999-bd22-cd10fc524542") : secret "llmisvc-webhook-server-cert" not found
Jul 13 20:25:21.756840 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.756814    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sj8r\" (UniqueName: \"kubernetes.io/projected/cbb99aa6-4589-4999-bd22-cd10fc524542-kube-api-access-4sj8r\") pod \"llmisvc-controller-manager-84fc959947-vwxlw\" (UID: \"cbb99aa6-4589-4999-bd22-cd10fc524542\") " pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:21.844834 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.844808    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vch\" (UniqueName: \"kubernetes.io/projected/4d64c846-a288-44f3-8857-a483217d8080-kube-api-access-n4vch\") pod \"seaweedfs-64568bcd49-tnckk\" (UID: \"4d64c846-a288-44f3-8857-a483217d8080\") " pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:21.844941 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.844887    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d64c846-a288-44f3-8857-a483217d8080-data\") pod \"seaweedfs-64568bcd49-tnckk\" (UID: \"4d64c846-a288-44f3-8857-a483217d8080\") " pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:21.845240 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.845223    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4d64c846-a288-44f3-8857-a483217d8080-data\") pod \"seaweedfs-64568bcd49-tnckk\" (UID: \"4d64c846-a288-44f3-8857-a483217d8080\") " pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:21.854328 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.854300    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vch\" (UniqueName: \"kubernetes.io/projected/4d64c846-a288-44f3-8857-a483217d8080-kube-api-access-n4vch\") pod \"seaweedfs-64568bcd49-tnckk\" (UID: \"4d64c846-a288-44f3-8857-a483217d8080\") " pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:21.940288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.940254    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:21.985782 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:21.985753    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:22.116512 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.116487    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-tnckk"]
Jul 13 20:25:22.117859 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:25:22.117833    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d64c846_a288_44f3_8857_a483217d8080.slice/crio-068d34b55284f47b1d0badcc3dbe28cc124ae70d5f653f5b0e2cd00aaf868fc9 WatchSource:0}: Error finding container 068d34b55284f47b1d0badcc3dbe28cc124ae70d5f653f5b0e2cd00aaf868fc9: Status 404 returned error can't find the container with id 068d34b55284f47b1d0badcc3dbe28cc124ae70d5f653f5b0e2cd00aaf868fc9
Jul 13 20:25:22.119082 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.119064    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:25:22.152103 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.152074    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:22.247303 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.247277    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbb99aa6-4589-4999-bd22-cd10fc524542-cert\") pod \"llmisvc-controller-manager-84fc959947-vwxlw\" (UID: \"cbb99aa6-4589-4999-bd22-cd10fc524542\") " pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:22.249522 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.249503    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbb99aa6-4589-4999-bd22-cd10fc524542-cert\") pod \"llmisvc-controller-manager-84fc959947-vwxlw\" (UID: \"cbb99aa6-4589-4999-bd22-cd10fc524542\") " pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:22.430166 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.430094    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tnckk" event={"ID":"4d64c846-a288-44f3-8857-a483217d8080","Type":"ContainerStarted","Data":"068d34b55284f47b1d0badcc3dbe28cc124ae70d5f653f5b0e2cd00aaf868fc9"}
Jul 13 20:25:22.532052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.532017    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:22.692715 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.692620    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-vwxlw"]
Jul 13 20:25:22.695986 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:25:22.695957    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcbb99aa6_4589_4999_bd22_cd10fc524542.slice/crio-b648924339c8a72d8cf4c8ba7c474e60554fb3f1c1530ecd40c2b44df28183c9 WatchSource:0}: Error finding container b648924339c8a72d8cf4c8ba7c474e60554fb3f1c1530ecd40c2b44df28183c9: Status 404 returned error can't find the container with id b648924339c8a72d8cf4c8ba7c474e60554fb3f1c1530ecd40c2b44df28183c9
Jul 13 20:25:22.908965 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:22.908933    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:23.011334 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:23.011246    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:23.076814 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:23.076772    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:23.435646 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:23.435606    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw" event={"ID":"cbb99aa6-4589-4999-bd22-cd10fc524542","Type":"ContainerStarted","Data":"b648924339c8a72d8cf4c8ba7c474e60554fb3f1c1530ecd40c2b44df28183c9"}
Jul 13 20:25:26.447507 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:26.447479    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw" event={"ID":"cbb99aa6-4589-4999-bd22-cd10fc524542","Type":"ContainerStarted","Data":"f77a702aed477b6f401b5a70bfa12f959f4e6af5f7774fc0eeb053d120f14c43"}
Jul 13 20:25:26.447826 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:26.447606    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:25:26.484993 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:26.484861    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw" podStartSLOduration=1.837043408 podStartE2EDuration="5.484840811s" totalImagesPullingTime="3.647797403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:25:26.484634574 +0000 UTC m=+383.956863351" watchObservedRunningTime="2026-07-13 20:25:26.484840811 +0000 UTC m=+383.957069589"
Jul 13 20:25:27.452737 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:27.452700    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-tnckk" event={"ID":"4d64c846-a288-44f3-8857-a483217d8080","Type":"ContainerStarted","Data":"899aadb9e7cea06ce0cdf8417fb206d36fa448bd62cff575750bf773a4de14c8"}
Jul 13 20:25:27.453204 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:27.452787    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:27.480446 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:27.480391    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-tnckk" podStartSLOduration=2.204287238 podStartE2EDuration="6.480371625s" totalImagesPullingTime="4.276084387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:25:27.478569749 +0000 UTC m=+384.950798519" watchObservedRunningTime="2026-07-13 20:25:27.480371625 +0000 UTC m=+384.952600401"
Jul 13 20:25:33.459229 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:33.459194    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-tnckk"
Jul 13 20:25:41.541797 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:41.541754    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:41.630484 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:41.630446    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:42.207519 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:42.207489    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:42.342601 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:42.342571    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:45.646860 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:45.646817    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:52.689619 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:52.689584    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:52.783854 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:52.783807    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:52.880784 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:52.880743    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:25:57.455285 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:25:57.455249    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-vwxlw"
Jul 13 20:26:32.477364 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.477323    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:32.530884 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.530850    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-6nh4z"]
Jul 13 20:26:32.533327 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.533304    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.536135 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.536112    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 13 20:26:32.536247 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.536190    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-mxcl2\""
Jul 13 20:26:32.553105 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.553082    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-6nh4z"]
Jul 13 20:26:32.573899 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.573846    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:32.650539 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.650509    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c-tls-certs\") pod \"model-serving-api-fd65d7d6-6nh4z\" (UID: \"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c\") " pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.650765 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.650558    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9m8\" (UniqueName: \"kubernetes.io/projected/9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c-kube-api-access-8h9m8\") pod \"model-serving-api-fd65d7d6-6nh4z\" (UID: \"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c\") " pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.664104 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.664075    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:32.751017 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.750923    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9m8\" (UniqueName: \"kubernetes.io/projected/9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c-kube-api-access-8h9m8\") pod \"model-serving-api-fd65d7d6-6nh4z\" (UID: \"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c\") " pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.751017 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.750982    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c-tls-certs\") pod \"model-serving-api-fd65d7d6-6nh4z\" (UID: \"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c\") " pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.753329 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.753296    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c-tls-certs\") pod \"model-serving-api-fd65d7d6-6nh4z\" (UID: \"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c\") " pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.764077 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.764054    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9m8\" (UniqueName: \"kubernetes.io/projected/9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c-kube-api-access-8h9m8\") pod \"model-serving-api-fd65d7d6-6nh4z\" (UID: \"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c\") " pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.833141 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.833085    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:32.843988 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.843962    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:32.975163 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:32.975131    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-6nh4z"]
Jul 13 20:26:32.980285 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:26:32.980263    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5fa0d7_2205_4f11_ae23_0eec5ffdcf3c.slice/crio-e99249b9a0f08203979be51c032bd5ae06fed7932cd1503b3daa373cecdb9e0a WatchSource:0}: Error finding container e99249b9a0f08203979be51c032bd5ae06fed7932cd1503b3daa373cecdb9e0a: Status 404 returned error can't find the container with id e99249b9a0f08203979be51c032bd5ae06fed7932cd1503b3daa373cecdb9e0a
Jul 13 20:26:33.311122 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:33.311091    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:33.375861 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:33.375826    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:26:33.670389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:33.670357    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-6nh4z" event={"ID":"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c","Type":"ContainerStarted","Data":"e99249b9a0f08203979be51c032bd5ae06fed7932cd1503b3daa373cecdb9e0a"}
Jul 13 20:26:35.679754 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:35.679711    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-6nh4z" event={"ID":"9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c","Type":"ContainerStarted","Data":"e4c5bcdabbdbc136ae8787555565c6d23a4d4776186df2b2be0e165374f760b6"}
Jul 13 20:26:35.680118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:35.679857    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:35.699043 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:35.698992    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-6nh4z" podStartSLOduration=1.1923903359999999 podStartE2EDuration="3.698978721s" totalImagesPullingTime="2.506588385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:26:35.697168167 +0000 UTC m=+453.169396935" watchObservedRunningTime="2026-07-13 20:26:35.698978721 +0000 UTC m=+453.171207495"
Jul 13 20:26:46.687047 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:46.687016    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-6nh4z"
Jul 13 20:26:49.549463 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.549430    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pkmb9"]
Jul 13 20:26:49.552455 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.552439    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pkmb9"
Jul 13 20:26:49.565509 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.565487    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pkmb9"]
Jul 13 20:26:49.669530 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.669497    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c09814fd-b698-4cbe-ab82-1db7f60a5212-model-cache\") pod \"s3-init-pkmb9\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") " pod="kserve/s3-init-pkmb9"
Jul 13 20:26:49.669738 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.669541    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqbn2\" (UniqueName: \"kubernetes.io/projected/c09814fd-b698-4cbe-ab82-1db7f60a5212-kube-api-access-fqbn2\") pod \"s3-init-pkmb9\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") " pod="kserve/s3-init-pkmb9"
Jul 13 20:26:49.770898 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.770864    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c09814fd-b698-4cbe-ab82-1db7f60a5212-model-cache\") pod \"s3-init-pkmb9\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") " pod="kserve/s3-init-pkmb9"
Jul 13 20:26:49.770898 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.770903    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbn2\" (UniqueName: \"kubernetes.io/projected/c09814fd-b698-4cbe-ab82-1db7f60a5212-kube-api-access-fqbn2\") pod \"s3-init-pkmb9\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") " pod="kserve/s3-init-pkmb9"
Jul 13 20:26:49.771310 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.771287    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c09814fd-b698-4cbe-ab82-1db7f60a5212-model-cache\") pod \"s3-init-pkmb9\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") " pod="kserve/s3-init-pkmb9"
Jul 13 20:26:49.780886 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.780856    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbn2\" (UniqueName: \"kubernetes.io/projected/c09814fd-b698-4cbe-ab82-1db7f60a5212-kube-api-access-fqbn2\") pod \"s3-init-pkmb9\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") " pod="kserve/s3-init-pkmb9"
Jul 13 20:26:49.861266 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:49.861222    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pkmb9"
Jul 13 20:26:50.001066 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:50.001036    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pkmb9"]
Jul 13 20:26:50.020237 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:26:50.020126    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc09814fd_b698_4cbe_ab82_1db7f60a5212.slice/crio-de6edc962e7d1c8241d4f5621606a56fbfa997f118b5bee31d68ea59510cc121 WatchSource:0}: Error finding container de6edc962e7d1c8241d4f5621606a56fbfa997f118b5bee31d68ea59510cc121: Status 404 returned error can't find the container with id de6edc962e7d1c8241d4f5621606a56fbfa997f118b5bee31d68ea59510cc121
Jul 13 20:26:50.733756 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:50.733724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pkmb9" event={"ID":"c09814fd-b698-4cbe-ab82-1db7f60a5212","Type":"ContainerStarted","Data":"de6edc962e7d1c8241d4f5621606a56fbfa997f118b5bee31d68ea59510cc121"}
Jul 13 20:26:54.750427 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:54.750382    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pkmb9" event={"ID":"c09814fd-b698-4cbe-ab82-1db7f60a5212","Type":"ContainerStarted","Data":"c9cd8708f0078cbf2d2a2034531b4807d89ea91baa2f83222681be4cc70c6032"}
Jul 13 20:26:55.756009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:55.755923    2551 generic.go:356] "Generic (PLEG): container finished" podID="c09814fd-b698-4cbe-ab82-1db7f60a5212" containerID="c9cd8708f0078cbf2d2a2034531b4807d89ea91baa2f83222681be4cc70c6032" exitCode=0
Jul 13 20:26:55.756009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:26:55.755955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pkmb9" event={"ID":"c09814fd-b698-4cbe-ab82-1db7f60a5212","Type":"ContainerDied","Data":"c9cd8708f0078cbf2d2a2034531b4807d89ea91baa2f83222681be4cc70c6032"}
Jul 13 20:27:00.777170 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:27:00.777079    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pkmb9" event={"ID":"c09814fd-b698-4cbe-ab82-1db7f60a5212","Type":"ContainerStarted","Data":"6358d098520171a27a34b9858bbeb3b5384d130f16782f016eca8107ca59075f"}
Jul 13 20:27:00.797156 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:27:00.797109    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-pkmb9" podStartSLOduration=1.810282338 podStartE2EDuration="11.797092788s" totalImagesPullingTime="8.923132356s" totalInitContainerRuntime="1.063678094s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:26:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:27:00.795057659 +0000 UTC m=+478.267286434" watchObservedRunningTime="2026-07-13 20:27:00.797092788 +0000 UTC m=+478.269321562"
Jul 13 20:27:06.577000 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:27:06.576959    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:27:26.713759 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:27:26.713717    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:06.000403 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:06.000373    2551 generic.go:356] "Generic (PLEG): container finished" podID="c09814fd-b698-4cbe-ab82-1db7f60a5212" containerID="6358d098520171a27a34b9858bbeb3b5384d130f16782f016eca8107ca59075f" exitCode=0
Jul 13 20:28:06.000837 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:06.000439    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pkmb9" event={"ID":"c09814fd-b698-4cbe-ab82-1db7f60a5212","Type":"ContainerDied","Data":"6358d098520171a27a34b9858bbeb3b5384d130f16782f016eca8107ca59075f"}
Jul 13 20:28:07.122529 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:07.122507    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pkmb9"
Jul 13 20:28:07.226399 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:07.226362    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqbn2\" (UniqueName: \"kubernetes.io/projected/c09814fd-b698-4cbe-ab82-1db7f60a5212-kube-api-access-fqbn2\") pod \"c09814fd-b698-4cbe-ab82-1db7f60a5212\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") "
Jul 13 20:28:07.226558 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:07.226414    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c09814fd-b698-4cbe-ab82-1db7f60a5212-model-cache\") pod \"c09814fd-b698-4cbe-ab82-1db7f60a5212\" (UID: \"c09814fd-b698-4cbe-ab82-1db7f60a5212\") "
Jul 13 20:28:07.228587 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:07.228563    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c09814fd-b698-4cbe-ab82-1db7f60a5212-kube-api-access-fqbn2" (OuterVolumeSpecName: "kube-api-access-fqbn2") pod "c09814fd-b698-4cbe-ab82-1db7f60a5212" (UID: "c09814fd-b698-4cbe-ab82-1db7f60a5212"). InnerVolumeSpecName "kube-api-access-fqbn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:28:07.244896 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:07.244851    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c09814fd-b698-4cbe-ab82-1db7f60a5212-model-cache" (OuterVolumeSpecName: "model-cache") pod "c09814fd-b698-4cbe-ab82-1db7f60a5212" (UID: "c09814fd-b698-4cbe-ab82-1db7f60a5212"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:28:07.327305 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:07.327237    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqbn2\" (UniqueName: \"kubernetes.io/projected/c09814fd-b698-4cbe-ab82-1db7f60a5212-kube-api-access-fqbn2\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:28:07.327305 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:07.327262    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c09814fd-b698-4cbe-ab82-1db7f60a5212-model-cache\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:28:08.008612 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:08.008573    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pkmb9" event={"ID":"c09814fd-b698-4cbe-ab82-1db7f60a5212","Type":"ContainerDied","Data":"de6edc962e7d1c8241d4f5621606a56fbfa997f118b5bee31d68ea59510cc121"}
Jul 13 20:28:08.008825 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:08.008625    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6edc962e7d1c8241d4f5621606a56fbfa997f118b5bee31d68ea59510cc121"
Jul 13 20:28:08.008825 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:08.008587    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pkmb9"
Jul 13 20:28:08.459803 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:08.459771    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:08.545424 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:08.545392    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:20.540325 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.540282    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:20.630618 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.630582    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"]
Jul 13 20:28:20.630963 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.630948    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09814fd-b698-4cbe-ab82-1db7f60a5212" containerName="download-hf-model"
Jul 13 20:28:20.631012 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.630967    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09814fd-b698-4cbe-ab82-1db7f60a5212" containerName="download-hf-model"
Jul 13 20:28:20.631012 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.630979    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c09814fd-b698-4cbe-ab82-1db7f60a5212" containerName="s3-init"
Jul 13 20:28:20.631012 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.630987    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c09814fd-b698-4cbe-ab82-1db7f60a5212" containerName="s3-init"
Jul 13 20:28:20.631099 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.631055    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c09814fd-b698-4cbe-ab82-1db7f60a5212" containerName="s3-init"
Jul 13 20:28:20.634625 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.634608    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.641990 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.641960    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 20:28:20.641990 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.641961    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-bede5-predictor-serving-cert\""
Jul 13 20:28:20.641990 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.641990    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 20:28:20.642236 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.642107    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-bede5-kube-rbac-proxy-sar-config\""
Jul 13 20:28:20.642356 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.642339    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 20:28:20.643148 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.643127    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:28:20.643355 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.643341    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d8krp\""
Jul 13 20:28:20.651512 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.651488    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"]
Jul 13 20:28:20.825349 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.825258    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8496\" (UniqueName: \"kubernetes.io/projected/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-api-access-m8496\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.825349 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.825309    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9d6485-196f-4f72-963d-8777d5ee8896-proxy-tls\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.825533 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.825364    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.825533 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.825425    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.825533 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.825464    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9d6485-196f-4f72-963d-8777d5ee8896-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.851082 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.851051    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:28:20.926367 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.926334    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.926523 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.926375    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9d6485-196f-4f72-963d-8777d5ee8896-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.926523 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.926407    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8496\" (UniqueName: \"kubernetes.io/projected/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-api-access-m8496\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.926523 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.926434    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9d6485-196f-4f72-963d-8777d5ee8896-proxy-tls\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.926523 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.926463    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.926913 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.926889    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9d6485-196f-4f72-963d-8777d5ee8896-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.927136 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.927119    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.927196 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.927178    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.928823 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.928806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9d6485-196f-4f72-963d-8777d5ee8896-proxy-tls\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:20.950846 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:20.950806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8496\" (UniqueName: \"kubernetes.io/projected/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-api-access-m8496\") pod \"isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:21.244542 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:21.244502    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:21.377919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:21.377806    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"]
Jul 13 20:28:21.380456 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:28:21.380431    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9d6485_196f_4f72_963d_8777d5ee8896.slice/crio-2df47903d8ae4ad076e49cdc9cd11a27db1c9063a8710b9de6bcb534f8df7810 WatchSource:0}: Error finding container 2df47903d8ae4ad076e49cdc9cd11a27db1c9063a8710b9de6bcb534f8df7810: Status 404 returned error can't find the container with id 2df47903d8ae4ad076e49cdc9cd11a27db1c9063a8710b9de6bcb534f8df7810
Jul 13 20:28:22.054565 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:22.054523    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerStarted","Data":"2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58"}
Jul 13 20:28:22.054993 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:22.054635    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerStarted","Data":"2df47903d8ae4ad076e49cdc9cd11a27db1c9063a8710b9de6bcb534f8df7810"}
Jul 13 20:28:26.070274 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:26.070238    2551 generic.go:356] "Generic (PLEG): container finished" podID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerID="2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58" exitCode=0
Jul 13 20:28:26.070692 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:26.070318    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerDied","Data":"2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58"}
Jul 13 20:28:41.141561 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:41.141521    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerStarted","Data":"eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73"}
Jul 13 20:28:43.149688 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:43.149590    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerStarted","Data":"9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15"}
Jul 13 20:28:46.163225 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:46.163179    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerStarted","Data":"35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab"}
Jul 13 20:28:46.163591 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:46.163409    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:46.163591 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:46.163552    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:46.164663 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:46.164629    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:28:46.192030 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:46.189308    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podStartSLOduration=2.883744518 podStartE2EDuration="26.189273568s" totalImagesPullingTime="19.556737161s" totalInitContainerRuntime="3.748791889s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:28:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:28:46.187009978 +0000 UTC m=+583.659238742" watchObservedRunningTime="2026-07-13 20:28:46.189273568 +0000 UTC m=+583.661502344"
Jul 13 20:28:47.166827 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:47.166786    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:47.167287 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:47.166982    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:28:47.168047 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:47.168019    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:28:47.170790 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:47.170773    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:28:48.170226 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:48.170184    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:28:48.170630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:48.170427    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:28:49.174008 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:49.173968    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:28:49.174395 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:49.174261    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:28:59.174783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:59.174738    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:28:59.175345 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:28:59.175237    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:29:02.954165 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:02.954136    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:29:02.954845 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:02.954823    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:29:09.174094 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:09.174044    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:29:09.305389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:09.174433    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:29:19.174393 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:19.174346    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:29:19.174897 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:19.174741    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:29:21.750441 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:21.750397    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:29:29.174357 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:29.174299    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:29:29.174783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:29.174711    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:29:39.174392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:39.174317    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:29:39.174927 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:39.174787    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:29:49.174610 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:49.174564    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:29:49.175113 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:49.175087    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:29:55.139296 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:55.139271    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:29:55.139694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:29:55.139348    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:30:05.728369 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.728331    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"]
Jul 13 20:30:05.728758 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.728710    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy" containerID="cri-o://9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15" gracePeriod=30
Jul 13 20:30:05.728833 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.728697    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" containerID="cri-o://35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab" gracePeriod=30
Jul 13 20:30:05.728930 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.728903    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" containerID="cri-o://eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73" gracePeriod=30
Jul 13 20:30:05.747473 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.747439    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:05.852553 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.852523    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"]
Jul 13 20:30:05.854825 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.854798    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.857102 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.857081    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-732aa-predictor-serving-cert\""
Jul 13 20:30:05.857206 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.857087    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-732aa-kube-rbac-proxy-sar-config\""
Jul 13 20:30:05.867085 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.867055    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"]
Jul 13 20:30:05.877266 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.877236    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c057fd-1612-4819-9f80-d40144c8c144-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.877423 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.877314    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgdc\" (UniqueName: \"kubernetes.io/projected/e6c057fd-1612-4819-9f80-d40144c8c144-kube-api-access-mbgdc\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.877423 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.877356    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.877423 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.877380    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-cabundle-cert\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.877423 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.877413    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.928431 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.928381    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:05.951157 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.951128    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"]
Jul 13 20:30:05.953493 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.953474    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:05.955877 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.955853    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-732aa-predictor-serving-cert\""
Jul 13 20:30:05.956114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.956097    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-732aa-kube-rbac-proxy-sar-config\""
Jul 13 20:30:05.967868 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.967846    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"]
Jul 13 20:30:05.977942 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.977918    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d99135-2d02-439a-a77b-19064b7f0244-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:05.978051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.977960    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-cabundle-cert\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:05.978051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.977995    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgdc\" (UniqueName: \"kubernetes.io/projected/e6c057fd-1612-4819-9f80-d40144c8c144-kube-api-access-mbgdc\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.978051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978033    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:05.978164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978059    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.978164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978084    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-cabundle-cert\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.978164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978106    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.978164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978126    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:05.978164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978149    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pfh\" (UniqueName: \"kubernetes.io/projected/12d99135-2d02-439a-a77b-19064b7f0244-kube-api-access-j6pfh\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:05.978345 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978182    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c057fd-1612-4819-9f80-d40144c8c144-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.978345 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:05.978217    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-serving-cert: secret "isvc-sklearn-graph-raw-732aa-predictor-serving-cert" not found
Jul 13 20:30:05.978345 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:05.978279    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls podName:e6c057fd-1612-4819-9f80-d40144c8c144 nodeName:}" failed. No retries permitted until 2026-07-13 20:30:06.478258094 +0000 UTC m=+663.950486861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls") pod "isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" (UID: "e6c057fd-1612-4819-9f80-d40144c8c144") : secret "isvc-sklearn-graph-raw-732aa-predictor-serving-cert" not found
Jul 13 20:30:05.978598 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978553    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c057fd-1612-4819-9f80-d40144c8c144-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.978994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978976    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-cabundle-cert\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.979066 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.978998    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:05.996261 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:05.996233    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgdc\" (UniqueName: \"kubernetes.io/projected/e6c057fd-1612-4819-9f80-d40144c8c144-kube-api-access-mbgdc\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:06.078725 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.078694    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-cabundle-cert\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.078895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.078747    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.078895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.078786    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.078895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.078803    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pfh\" (UniqueName: \"kubernetes.io/projected/12d99135-2d02-439a-a77b-19064b7f0244-kube-api-access-j6pfh\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.078895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.078850    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d99135-2d02-439a-a77b-19064b7f0244-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.079247 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:06.078929    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-serving-cert: secret "isvc-xgboost-graph-raw-732aa-predictor-serving-cert" not found
Jul 13 20:30:06.079247 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:06.078993    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls podName:12d99135-2d02-439a-a77b-19064b7f0244 nodeName:}" failed. No retries permitted until 2026-07-13 20:30:06.578975539 +0000 UTC m=+664.051204295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls") pod "isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" (UID: "12d99135-2d02-439a-a77b-19064b7f0244") : secret "isvc-xgboost-graph-raw-732aa-predictor-serving-cert" not found
Jul 13 20:30:06.079247 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.079236    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d99135-2d02-439a-a77b-19064b7f0244-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.079421 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.079394    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-cabundle-cert\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.079578 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.079553    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.087785 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.087764    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pfh\" (UniqueName: \"kubernetes.io/projected/12d99135-2d02-439a-a77b-19064b7f0244-kube-api-access-j6pfh\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.164963 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.164914    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:06.225217 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.225183    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:06.427060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.427021    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:06.442867 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.442840    2551 generic.go:356] "Generic (PLEG): container finished" podID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerID="9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15" exitCode=2
Jul 13 20:30:06.442996 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.442908    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerDied","Data":"9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15"}
Jul 13 20:30:06.483460 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.483429    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:06.485856 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.485838    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls\") pod \"isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:06.584171 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.584135    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.586508 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.586485    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls\") pod \"isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.766289 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.766205    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:06.864060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.864023    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:06.897154 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:06.897122    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"]
Jul 13 20:30:06.900500 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:30:06.900467    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c057fd_1612_4819_9f80_d40144c8c144.slice/crio-cb19c4887096f2899a15b211b44fd97f312e0c2ec342340960fa82662135bd97 WatchSource:0}: Error finding container cb19c4887096f2899a15b211b44fd97f312e0c2ec342340960fa82662135bd97: Status 404 returned error can't find the container with id cb19c4887096f2899a15b211b44fd97f312e0c2ec342340960fa82662135bd97
Jul 13 20:30:07.002282 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:07.002248    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"]
Jul 13 20:30:07.005169 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:30:07.005145    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d99135_2d02_439a_a77b_19064b7f0244.slice/crio-01e60cd24b0fc157f56eedb834edfec89f7435dd6fa941b2248c66577ce3ff07 WatchSource:0}: Error finding container 01e60cd24b0fc157f56eedb834edfec89f7435dd6fa941b2248c66577ce3ff07: Status 404 returned error can't find the container with id 01e60cd24b0fc157f56eedb834edfec89f7435dd6fa941b2248c66577ce3ff07
Jul 13 20:30:07.167383 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:07.167339    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 13 20:30:07.449113 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:07.449077    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerStarted","Data":"191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0"}
Jul 13 20:30:07.449272 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:07.449144    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerStarted","Data":"cb19c4887096f2899a15b211b44fd97f312e0c2ec342340960fa82662135bd97"}
Jul 13 20:30:07.450849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:07.450818    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerStarted","Data":"c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31"}
Jul 13 20:30:07.450963 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:07.450884    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerStarted","Data":"01e60cd24b0fc157f56eedb834edfec89f7435dd6fa941b2248c66577ce3ff07"}
Jul 13 20:30:10.464683 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:10.464637    2551 generic.go:356] "Generic (PLEG): container finished" podID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerID="eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73" exitCode=0
Jul 13 20:30:10.465065 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:10.464713    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerDied","Data":"eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73"}
Jul 13 20:30:11.469557 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:11.469526    2551 generic.go:356] "Generic (PLEG): container finished" podID="e6c057fd-1612-4819-9f80-d40144c8c144" containerID="191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0" exitCode=0
Jul 13 20:30:11.470025 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:11.469594    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerDied","Data":"191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0"}
Jul 13 20:30:11.471312 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:11.471269    2551 generic.go:356] "Generic (PLEG): container finished" podID="12d99135-2d02-439a-a77b-19064b7f0244" containerID="c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31" exitCode=0
Jul 13 20:30:11.471312 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:11.471300    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerDied","Data":"c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31"}
Jul 13 20:30:12.167867 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:12.167815    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 13 20:30:12.479592 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:12.479513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerStarted","Data":"8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec"}
Jul 13 20:30:12.480035 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:12.479634    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerStarted","Data":"136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321"}
Jul 13 20:30:12.480107 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:12.480083    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:12.480323 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:12.480300    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:12.481702 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:12.481668    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:30:12.501117 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:12.501019    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podStartSLOduration=3.882144252 podStartE2EDuration="7.500999439s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.618855187s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:30:12.500286165 +0000 UTC m=+669.972514935" watchObservedRunningTime="2026-07-13 20:30:12.500999439 +0000 UTC m=+669.973228212"
Jul 13 20:30:13.484835 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:13.484789    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:30:14.488756 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:14.488718    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:30:15.132237 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:15.132178    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:30:15.132500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:15.132469    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:30:17.167086 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:17.167043    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 13 20:30:17.167455 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:17.167178    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:30:17.416145 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:17.416106    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:17.498077 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:17.497986    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:30:19.493881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:19.493854    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:30:19.494425 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:19.494398    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:30:22.167992 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:22.167939    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 13 20:30:25.132513 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:25.132467    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:30:25.133000 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:25.132870    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:30:27.167843 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:27.167806    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 13 20:30:29.494461 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:29.494421    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:30:32.167640 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:32.167598    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 13 20:30:32.561967 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:32.561938    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerStarted","Data":"d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907"}
Jul 13 20:30:33.568496 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:33.568455    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerStarted","Data":"9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec"}
Jul 13 20:30:33.568912 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:33.568688    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:33.590253 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:33.590162    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podStartSLOduration=3.6775747770000002 podStartE2EDuration="28.590144067s" totalImagesPullingTime="20.969300939s" totalInitContainerRuntime="3.943268351s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:30:33.589986181 +0000 UTC m=+691.062214960" watchObservedRunningTime="2026-07-13 20:30:33.590144067 +0000 UTC m=+691.062372844"
Jul 13 20:30:34.572701 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:34.572660    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:34.573888 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:34.573859    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:30:35.132098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.132060    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 13 20:30:35.132431 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.132399    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:30:35.137307 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.137283    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:30:35.137427 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.137387    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:30:35.576525 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.576439    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:30:35.872211 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.872186    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:30:35.942291 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942261    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9d6485-196f-4f72-963d-8777d5ee8896-kserve-provision-location\") pod \"3e9d6485-196f-4f72-963d-8777d5ee8896\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") "
Jul 13 20:30:35.942453 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942303    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9d6485-196f-4f72-963d-8777d5ee8896-proxy-tls\") pod \"3e9d6485-196f-4f72-963d-8777d5ee8896\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") "
Jul 13 20:30:35.942453 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942323    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-cabundle-cert\") pod \"3e9d6485-196f-4f72-963d-8777d5ee8896\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") "
Jul 13 20:30:35.942453 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942359    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8496\" (UniqueName: \"kubernetes.io/projected/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-api-access-m8496\") pod \"3e9d6485-196f-4f72-963d-8777d5ee8896\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") "
Jul 13 20:30:35.942453 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942387    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-rbac-proxy-sar-config\") pod \"3e9d6485-196f-4f72-963d-8777d5ee8896\" (UID: \"3e9d6485-196f-4f72-963d-8777d5ee8896\") "
Jul 13 20:30:35.942689 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942638    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e9d6485-196f-4f72-963d-8777d5ee8896-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e9d6485-196f-4f72-963d-8777d5ee8896" (UID: "3e9d6485-196f-4f72-963d-8777d5ee8896"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:30:35.942814 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942785    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e9d6485-196f-4f72-963d-8777d5ee8896" (UID: "3e9d6485-196f-4f72-963d-8777d5ee8896"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:30:35.942882 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.942813    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e9d6485-196f-4f72-963d-8777d5ee8896" (UID: "3e9d6485-196f-4f72-963d-8777d5ee8896"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:30:35.944451 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.944425    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9d6485-196f-4f72-963d-8777d5ee8896-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e9d6485-196f-4f72-963d-8777d5ee8896" (UID: "3e9d6485-196f-4f72-963d-8777d5ee8896"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:30:35.944550 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:35.944483    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-api-access-m8496" (OuterVolumeSpecName: "kube-api-access-m8496") pod "3e9d6485-196f-4f72-963d-8777d5ee8896" (UID: "3e9d6485-196f-4f72-963d-8777d5ee8896"). InnerVolumeSpecName "kube-api-access-m8496". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:30:36.043466 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.043431    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e9d6485-196f-4f72-963d-8777d5ee8896-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:30:36.043466 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.043458    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e9d6485-196f-4f72-963d-8777d5ee8896-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:30:36.043466 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.043469    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:30:36.043466 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.043478    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8496\" (UniqueName: \"kubernetes.io/projected/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-api-access-m8496\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:30:36.043731 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.043488    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e9d6485-196f-4f72-963d-8777d5ee8896-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:30:36.582527 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.582493    2551 generic.go:356] "Generic (PLEG): container finished" podID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerID="35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab" exitCode=0
Jul 13 20:30:36.582955 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.582553    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"
Jul 13 20:30:36.582955 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.582567    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerDied","Data":"35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab"}
Jul 13 20:30:36.582955 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.582631    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp" event={"ID":"3e9d6485-196f-4f72-963d-8777d5ee8896","Type":"ContainerDied","Data":"2df47903d8ae4ad076e49cdc9cd11a27db1c9063a8710b9de6bcb534f8df7810"}
Jul 13 20:30:36.582955 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.582683    2551 scope.go:117] "RemoveContainer" containerID="35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab"
Jul 13 20:30:36.590878 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.590858    2551 scope.go:117] "RemoveContainer" containerID="9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15"
Jul 13 20:30:36.597895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.597880    2551 scope.go:117] "RemoveContainer" containerID="eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73"
Jul 13 20:30:36.604324 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.604305    2551 scope.go:117] "RemoveContainer" containerID="2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58"
Jul 13 20:30:36.612169 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.607619    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"]
Jul 13 20:30:36.612169 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.610628    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-bede5-predictor-bddd8fd8b-hcpfp"]
Jul 13 20:30:36.615765 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.615741    2551 scope.go:117] "RemoveContainer" containerID="35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab"
Jul 13 20:30:36.616036 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:36.616017    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab\": container with ID starting with 35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab not found: ID does not exist" containerID="35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab"
Jul 13 20:30:36.616114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.616047    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab"} err="failed to get container status \"35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab\": rpc error: code = NotFound desc = could not find container \"35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab\": container with ID starting with 35ef56d0c8ca4ba59fd7e63a0c9e987bf8611b332a2a9bbe62b3b58ad19272ab not found: ID does not exist"
Jul 13 20:30:36.616114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.616073    2551 scope.go:117] "RemoveContainer" containerID="9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15"
Jul 13 20:30:36.616399 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:36.616379    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15\": container with ID starting with 9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15 not found: ID does not exist" containerID="9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15"
Jul 13 20:30:36.616439 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.616407    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15"} err="failed to get container status \"9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15\": rpc error: code = NotFound desc = could not find container \"9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15\": container with ID starting with 9101e56ffb8796fb73919692667f07c87947e55a3f2285c407100027036b2e15 not found: ID does not exist"
Jul 13 20:30:36.616439 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.616424    2551 scope.go:117] "RemoveContainer" containerID="eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73"
Jul 13 20:30:36.616669 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:36.616631    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73\": container with ID starting with eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73 not found: ID does not exist" containerID="eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73"
Jul 13 20:30:36.616781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.616754    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73"} err="failed to get container status \"eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73\": rpc error: code = NotFound desc = could not find container \"eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73\": container with ID starting with eac9f4d5c616f3cf2e82c3a754edfaa2afec95a19e331c8e9f0612437a91ff73 not found: ID does not exist"
Jul 13 20:30:36.616862 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.616785    2551 scope.go:117] "RemoveContainer" containerID="2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58"
Jul 13 20:30:36.617034 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:30:36.617015    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58\": container with ID starting with 2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58 not found: ID does not exist" containerID="2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58"
Jul 13 20:30:36.617083 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:36.617043    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58"} err="failed to get container status \"2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58\": rpc error: code = NotFound desc = could not find container \"2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58\": container with ID starting with 2d7c02e87311deedb1eae15b12534bb11645e9deb4a59b7efab0710cfc0d9c58 not found: ID does not exist"
Jul 13 20:30:37.136133 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:37.136103    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" path="/var/lib/kubelet/pods/3e9d6485-196f-4f72-963d-8777d5ee8896/volumes"
Jul 13 20:30:39.494895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:39.494853    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:30:40.581594 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:40.581564    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:30:40.582102 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:40.582078    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:30:49.494756 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:49.494711    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:30:50.582601 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:50.582565    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:30:59.494944 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:30:59.494859    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:31:00.582453 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:00.582413    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:31:09.494685 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:09.494628    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:31:10.582536 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:10.582496    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:31:19.495264 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:19.495223    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:31:20.582418 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:20.582380    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:31:29.495815 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:29.495781    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:31:30.582726 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:30.582683    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Jul 13 20:31:40.583431 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:40.583402    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:31:45.896300 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:45.896269    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:46.259276 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:46.259187    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:55.943374 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:55.943330    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:56.112804 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.112767    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"]
Jul 13 20:31:56.113128 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.113103    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" containerID="cri-o://136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321" gracePeriod=30
Jul 13 20:31:56.113192 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.113119    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kube-rbac-proxy" containerID="cri-o://8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec" gracePeriod=30
Jul 13 20:31:56.120003 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.119969    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:56.186862 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.186823    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:56.192016 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.191993    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"]
Jul 13 20:31:56.192277 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192266    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container"
Jul 13 20:31:56.192317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192278    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container"
Jul 13 20:31:56.192317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192290    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent"
Jul 13 20:31:56.192317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192296    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent"
Jul 13 20:31:56.192317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192302    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="storage-initializer"
Jul 13 20:31:56.192317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192308    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="storage-initializer"
Jul 13 20:31:56.192317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192314    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy"
Jul 13 20:31:56.192317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192320    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy"
Jul 13 20:31:56.192510 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192363    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kserve-container"
Jul 13 20:31:56.192510 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192371    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="kube-rbac-proxy"
Jul 13 20:31:56.192510 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.192376    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9d6485-196f-4f72-963d-8777d5ee8896" containerName="agent"
Jul 13 20:31:56.194478 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.194421    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.196995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.196970    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-553c9-kube-rbac-proxy-sar-config\""
Jul 13 20:31:56.196995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.196990    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-553c9-predictor-serving-cert\""
Jul 13 20:31:56.208303 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.208280    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"]
Jul 13 20:31:56.283379 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.283345    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:56.284126 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.284105    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"]
Jul 13 20:31:56.286398 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.286377    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.288537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.288515    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-553c9-predictor-serving-cert\""
Jul 13 20:31:56.288870 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.288851    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-553c9-kube-rbac-proxy-sar-config\""
Jul 13 20:31:56.303909 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.303888    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"]
Jul 13 20:31:56.340174 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.340145    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.340174 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.340178    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037c5f99-8d01-4345-97e9-01c25359948f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.340354 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.340210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.340354 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.340291    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037c5f99-8d01-4345-97e9-01c25359948f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.340354 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.340323    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xc2\" (UniqueName: \"kubernetes.io/projected/037c5f99-8d01-4345-97e9-01c25359948f-kube-api-access-m8xc2\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.353588 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.353560    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"]
Jul 13 20:31:56.353829 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.353806    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container" containerID="cri-o://d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907" gracePeriod=30
Jul 13 20:31:56.353905 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.353827    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kube-rbac-proxy" containerID="cri-o://9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec" gracePeriod=30
Jul 13 20:31:56.360057 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.360010    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:56.426832 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.426778    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:56.441624 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.441597    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmlc\" (UniqueName: \"kubernetes.io/projected/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-api-access-gxmlc\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.441774 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.441645    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.441774 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.441718    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037c5f99-8d01-4345-97e9-01c25359948f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.441856 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.441808    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.441898 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.441866    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a595e79-b7c7-48ea-8280-abe131fa3159-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.441947 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.441897    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a595e79-b7c7-48ea-8280-abe131fa3159-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.441999 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.441958    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037c5f99-8d01-4345-97e9-01c25359948f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.442052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.442005    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.442052 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.442044    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xc2\" (UniqueName: \"kubernetes.io/projected/037c5f99-8d01-4345-97e9-01c25359948f-kube-api-access-m8xc2\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.442150 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.442088    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.442150 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.442097    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037c5f99-8d01-4345-97e9-01c25359948f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.442538 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.442516    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.442587 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.442516    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.444552 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.444504    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037c5f99-8d01-4345-97e9-01c25359948f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.450155 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.450131    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xc2\" (UniqueName: \"kubernetes.io/projected/037c5f99-8d01-4345-97e9-01c25359948f-kube-api-access-m8xc2\") pod \"isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.508556 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.508522    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:31:56.542523 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.542487    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a595e79-b7c7-48ea-8280-abe131fa3159-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.542523 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.542531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a595e79-b7c7-48ea-8280-abe131fa3159-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.542768 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.542562    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.542768 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.542599    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.542768 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.542628    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmlc\" (UniqueName: \"kubernetes.io/projected/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-api-access-gxmlc\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.543204 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.543160    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a595e79-b7c7-48ea-8280-abe131fa3159-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.543321 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.543306    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.543382 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.543335    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.544981 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.544952    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a595e79-b7c7-48ea-8280-abe131fa3159-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.547326 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.547299    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:31:56.550432 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.550410    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmlc\" (UniqueName: \"kubernetes.io/projected/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-api-access-gxmlc\") pod \"isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.597261 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.597232    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:31:56.637158 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.637121    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"]
Jul 13 20:31:56.640450 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:31:56.640372    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037c5f99_8d01_4345_97e9_01c25359948f.slice/crio-3507b85e1bda78938c3305c036d4543dc977ea806d3ffa7b17b9379c2121af29 WatchSource:0}: Error finding container 3507b85e1bda78938c3305c036d4543dc977ea806d3ffa7b17b9379c2121af29: Status 404 returned error can't find the container with id 3507b85e1bda78938c3305c036d4543dc977ea806d3ffa7b17b9379c2121af29
Jul 13 20:31:56.733819 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.733791    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"]
Jul 13 20:31:56.735952 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:31:56.735928    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a595e79_b7c7_48ea_8280_abe131fa3159.slice/crio-d2caab08206007f073fc9dd03c15695f492a28fda457eb0c0c8b678d16fb618e WatchSource:0}: Error finding container d2caab08206007f073fc9dd03c15695f492a28fda457eb0c0c8b678d16fb618e: Status 404 returned error can't find the container with id d2caab08206007f073fc9dd03c15695f492a28fda457eb0c0c8b678d16fb618e
Jul 13 20:31:56.874510 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.874473    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerStarted","Data":"39bb87df71f554c11b92bf042ea9a2b96f6ee4484da5c94d1cf64a16730b92f1"}
Jul 13 20:31:56.874727 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.874541    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerStarted","Data":"d2caab08206007f073fc9dd03c15695f492a28fda457eb0c0c8b678d16fb618e"}
Jul 13 20:31:56.876801 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.876776    2551 generic.go:356] "Generic (PLEG): container finished" podID="e6c057fd-1612-4819-9f80-d40144c8c144" containerID="8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec" exitCode=2
Jul 13 20:31:56.876931 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.876852    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerDied","Data":"8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec"}
Jul 13 20:31:56.879148 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.879126    2551 generic.go:356] "Generic (PLEG): container finished" podID="12d99135-2d02-439a-a77b-19064b7f0244" containerID="9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec" exitCode=2
Jul 13 20:31:56.879249 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.879187    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerDied","Data":"9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec"}
Jul 13 20:31:56.880853 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.880831    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerStarted","Data":"a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6"}
Jul 13 20:31:56.880978 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:56.880873    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerStarted","Data":"3507b85e1bda78938c3305c036d4543dc977ea806d3ffa7b17b9379c2121af29"}
Jul 13 20:31:59.489139 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:59.489091    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 13 20:31:59.494495 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:31:59.494473    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Jul 13 20:32:00.105506 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.105482    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:32:00.277143 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277053    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d99135-2d02-439a-a77b-19064b7f0244-kserve-provision-location\") pod \"12d99135-2d02-439a-a77b-19064b7f0244\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") "
Jul 13 20:32:00.277143 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277117    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6pfh\" (UniqueName: \"kubernetes.io/projected/12d99135-2d02-439a-a77b-19064b7f0244-kube-api-access-j6pfh\") pod \"12d99135-2d02-439a-a77b-19064b7f0244\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") "
Jul 13 20:32:00.277358 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277148    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls\") pod \"12d99135-2d02-439a-a77b-19064b7f0244\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") "
Jul 13 20:32:00.277358 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277187    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-cabundle-cert\") pod \"12d99135-2d02-439a-a77b-19064b7f0244\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") "
Jul 13 20:32:00.277358 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277228    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-kube-rbac-proxy-sar-config\") pod \"12d99135-2d02-439a-a77b-19064b7f0244\" (UID: \"12d99135-2d02-439a-a77b-19064b7f0244\") "
Jul 13 20:32:00.277503 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277376    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d99135-2d02-439a-a77b-19064b7f0244-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12d99135-2d02-439a-a77b-19064b7f0244" (UID: "12d99135-2d02-439a-a77b-19064b7f0244"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:32:00.277503 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277483    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12d99135-2d02-439a-a77b-19064b7f0244-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:00.277605 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277563    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12d99135-2d02-439a-a77b-19064b7f0244" (UID: "12d99135-2d02-439a-a77b-19064b7f0244"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:32:00.277741 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.277711    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12d99135-2d02-439a-a77b-19064b7f0244" (UID: "12d99135-2d02-439a-a77b-19064b7f0244"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:32:00.279472 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.279441    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d99135-2d02-439a-a77b-19064b7f0244-kube-api-access-j6pfh" (OuterVolumeSpecName: "kube-api-access-j6pfh") pod "12d99135-2d02-439a-a77b-19064b7f0244" (UID: "12d99135-2d02-439a-a77b-19064b7f0244"). InnerVolumeSpecName "kube-api-access-j6pfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:32:00.279472 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.279443    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12d99135-2d02-439a-a77b-19064b7f0244" (UID: "12d99135-2d02-439a-a77b-19064b7f0244"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:32:00.378156 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.378118    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6pfh\" (UniqueName: \"kubernetes.io/projected/12d99135-2d02-439a-a77b-19064b7f0244-kube-api-access-j6pfh\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:00.378156 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.378146    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12d99135-2d02-439a-a77b-19064b7f0244-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:00.378156 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.378158    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:00.378156 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.378167    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12d99135-2d02-439a-a77b-19064b7f0244-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:00.898271 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.898241    2551 generic.go:356] "Generic (PLEG): container finished" podID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerID="39bb87df71f554c11b92bf042ea9a2b96f6ee4484da5c94d1cf64a16730b92f1" exitCode=0
Jul 13 20:32:00.898673 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.898315    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerDied","Data":"39bb87df71f554c11b92bf042ea9a2b96f6ee4484da5c94d1cf64a16730b92f1"}
Jul 13 20:32:00.900284 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.900261    2551 generic.go:356] "Generic (PLEG): container finished" podID="12d99135-2d02-439a-a77b-19064b7f0244" containerID="d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907" exitCode=0
Jul 13 20:32:00.900384 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.900328    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"
Jul 13 20:32:00.900384 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.900335    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerDied","Data":"d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907"}
Jul 13 20:32:00.900488 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.900385    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67" event={"ID":"12d99135-2d02-439a-a77b-19064b7f0244","Type":"ContainerDied","Data":"01e60cd24b0fc157f56eedb834edfec89f7435dd6fa941b2248c66577ce3ff07"}
Jul 13 20:32:00.900488 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.900419    2551 scope.go:117] "RemoveContainer" containerID="9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec"
Jul 13 20:32:00.902035 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.902018    2551 generic.go:356] "Generic (PLEG): container finished" podID="037c5f99-8d01-4345-97e9-01c25359948f" containerID="a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6" exitCode=0
Jul 13 20:32:00.902128 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.902092    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerDied","Data":"a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6"}
Jul 13 20:32:00.925518 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.925489    2551 scope.go:117] "RemoveContainer" containerID="d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907"
Jul 13 20:32:00.935518 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.935492    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"]
Jul 13 20:32:00.941194 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.941163    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-732aa-predictor-57ffffc8f7-lll67"]
Jul 13 20:32:00.956676 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.952763    2551 scope.go:117] "RemoveContainer" containerID="c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31"
Jul 13 20:32:00.970882 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.970681    2551 scope.go:117] "RemoveContainer" containerID="9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec"
Jul 13 20:32:00.971040 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:32:00.971018    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec\": container with ID starting with 9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec not found: ID does not exist" containerID="9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec"
Jul 13 20:32:00.971107 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.971047    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec"} err="failed to get container status \"9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec\": rpc error: code = NotFound desc = could not find container \"9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec\": container with ID starting with 9be3a3db7ab668d58af0f20e944c52071b4d19aa02de7c8c6ddb51f073bfd5ec not found: ID does not exist"
Jul 13 20:32:00.971107 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.971065    2551 scope.go:117] "RemoveContainer" containerID="d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907"
Jul 13 20:32:00.971310 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:32:00.971290    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907\": container with ID starting with d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907 not found: ID does not exist" containerID="d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907"
Jul 13 20:32:00.971363 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.971313    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907"} err="failed to get container status \"d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907\": rpc error: code = NotFound desc = could not find container \"d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907\": container with ID starting with d78fefcfc1bdbf0be0816f06001782b8accc555b257a5d902a4c91f893c78907 not found: ID does not exist"
Jul 13 20:32:00.971363 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.971326    2551 scope.go:117] "RemoveContainer" containerID="c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31"
Jul 13 20:32:00.971586 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:32:00.971557    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31\": container with ID starting with c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31 not found: ID does not exist" containerID="c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31"
Jul 13 20:32:00.971668 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:00.971598    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31"} err="failed to get container status \"c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31\": rpc error: code = NotFound desc = could not find container \"c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31\": container with ID starting with c5d5e294f9b899816001ac8d43ca3e8d9706bf853ead85970cbe90cd51fbad31 not found: ID does not exist"
Jul 13 20:32:01.075562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.075510    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:32:01.136520 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.136490    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12d99135-2d02-439a-a77b-19064b7f0244" path="/var/lib/kubelet/pods/12d99135-2d02-439a-a77b-19064b7f0244/volumes"
Jul 13 20:32:01.184261 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184227    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-kube-rbac-proxy-sar-config\") pod \"e6c057fd-1612-4819-9f80-d40144c8c144\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") "
Jul 13 20:32:01.184261 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184265    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbgdc\" (UniqueName: \"kubernetes.io/projected/e6c057fd-1612-4819-9f80-d40144c8c144-kube-api-access-mbgdc\") pod \"e6c057fd-1612-4819-9f80-d40144c8c144\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") "
Jul 13 20:32:01.184481 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184302    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-cabundle-cert\") pod \"e6c057fd-1612-4819-9f80-d40144c8c144\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") "
Jul 13 20:32:01.184481 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184330    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c057fd-1612-4819-9f80-d40144c8c144-kserve-provision-location\") pod \"e6c057fd-1612-4819-9f80-d40144c8c144\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") "
Jul 13 20:32:01.184481 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184376    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls\") pod \"e6c057fd-1612-4819-9f80-d40144c8c144\" (UID: \"e6c057fd-1612-4819-9f80-d40144c8c144\") "
Jul 13 20:32:01.184674 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184623    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c057fd-1612-4819-9f80-d40144c8c144-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6c057fd-1612-4819-9f80-d40144c8c144" (UID: "e6c057fd-1612-4819-9f80-d40144c8c144"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:32:01.184753 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184642    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e6c057fd-1612-4819-9f80-d40144c8c144" (UID: "e6c057fd-1612-4819-9f80-d40144c8c144"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:32:01.184753 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.184676    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e6c057fd-1612-4819-9f80-d40144c8c144" (UID: "e6c057fd-1612-4819-9f80-d40144c8c144"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:32:01.186423 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.186403    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c057fd-1612-4819-9f80-d40144c8c144-kube-api-access-mbgdc" (OuterVolumeSpecName: "kube-api-access-mbgdc") pod "e6c057fd-1612-4819-9f80-d40144c8c144" (UID: "e6c057fd-1612-4819-9f80-d40144c8c144"). InnerVolumeSpecName "kube-api-access-mbgdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:32:01.186494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.186453    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e6c057fd-1612-4819-9f80-d40144c8c144" (UID: "e6c057fd-1612-4819-9f80-d40144c8c144"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:32:01.285477 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.285448    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6c057fd-1612-4819-9f80-d40144c8c144-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:01.285477 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.285474    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:01.285477 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.285484    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbgdc\" (UniqueName: \"kubernetes.io/projected/e6c057fd-1612-4819-9f80-d40144c8c144-kube-api-access-mbgdc\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:01.285719 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.285493    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e6c057fd-1612-4819-9f80-d40144c8c144-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:01.285719 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.285502    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6c057fd-1612-4819-9f80-d40144c8c144-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:32:01.908214 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.908178    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerStarted","Data":"9c429734086454a163428dca2a8a2f1d2a5272f76449e6daab90f78243684391"}
Jul 13 20:32:01.908626 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.908251    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerStarted","Data":"c7915b2279e45c5c9c674582c59c6690dad5c16af258daf9110d5c778b340cb5"}
Jul 13 20:32:01.908626 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.908473    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:32:01.909951 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.909928    2551 generic.go:356] "Generic (PLEG): container finished" podID="e6c057fd-1612-4819-9f80-d40144c8c144" containerID="136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321" exitCode=0
Jul 13 20:32:01.910041 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.909976    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"
Jul 13 20:32:01.910041 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.910002    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerDied","Data":"136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321"}
Jul 13 20:32:01.910138 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.910066    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg" event={"ID":"e6c057fd-1612-4819-9f80-d40144c8c144","Type":"ContainerDied","Data":"cb19c4887096f2899a15b211b44fd97f312e0c2ec342340960fa82662135bd97"}
Jul 13 20:32:01.910138 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.910109    2551 scope.go:117] "RemoveContainer" containerID="8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec"
Jul 13 20:32:01.913078 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.913051    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerStarted","Data":"dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f"}
Jul 13 20:32:01.913160 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.913118    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerStarted","Data":"d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f"}
Jul 13 20:32:01.913389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.913375    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:32:01.913460 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.913436    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:32:01.914891 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.914870    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:01.918688 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.918669    2551 scope.go:117] "RemoveContainer" containerID="136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321"
Jul 13 20:32:01.926494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.926478    2551 scope.go:117] "RemoveContainer" containerID="191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0"
Jul 13 20:32:01.933620 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.933602    2551 scope.go:117] "RemoveContainer" containerID="8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec"
Jul 13 20:32:01.933918 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:32:01.933898    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec\": container with ID starting with 8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec not found: ID does not exist" containerID="8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec"
Jul 13 20:32:01.934003 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.933929    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec"} err="failed to get container status \"8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec\": rpc error: code = NotFound desc = could not find container \"8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec\": container with ID starting with 8b1ca809abbca748918b5d787ff23529d353c52bd876c84a0976d728b57957ec not found: ID does not exist"
Jul 13 20:32:01.934003 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.933961    2551 scope.go:117] "RemoveContainer" containerID="136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321"
Jul 13 20:32:01.934092 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.934004    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podStartSLOduration=2.265091766 podStartE2EDuration="5.933991092s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.668899326s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:32:01.931021527 +0000 UTC m=+779.403250300" watchObservedRunningTime="2026-07-13 20:32:01.933991092 +0000 UTC m=+779.406219866"
Jul 13 20:32:01.934240 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:32:01.934222    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321\": container with ID starting with 136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321 not found: ID does not exist" containerID="136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321"
Jul 13 20:32:01.934312 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.934247    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321"} err="failed to get container status \"136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321\": rpc error: code = NotFound desc = could not find container \"136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321\": container with ID starting with 136684905cc7e68060c92e1f7603ce68c54fcbaab49fa305739ef33d33cec321 not found: ID does not exist"
Jul 13 20:32:01.934312 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.934266    2551 scope.go:117] "RemoveContainer" containerID="191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0"
Jul 13 20:32:01.934519 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:32:01.934501    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0\": container with ID starting with 191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0 not found: ID does not exist" containerID="191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0"
Jul 13 20:32:01.934566 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.934525    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0"} err="failed to get container status \"191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0\": rpc error: code = NotFound desc = could not find container \"191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0\": container with ID starting with 191a42198f723fb5bac59dea6fd0c9fbef375611d5e80a9fff03697f35cf16a0 not found: ID does not exist"
Jul 13 20:32:01.946617 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.946598    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"]
Jul 13 20:32:01.950598 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.950576    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-732aa-predictor-76df45c78-k9wzg"]
Jul 13 20:32:01.975119 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:01.975075    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podStartSLOduration=2.203472713 podStartE2EDuration="5.975061563s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.77158885s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:31:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:32:01.974575663 +0000 UTC m=+779.446804459" watchObservedRunningTime="2026-07-13 20:32:01.975061563 +0000 UTC m=+779.447290337"
Jul 13 20:32:02.917804 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:02.917768    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:32:02.918274 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:02.917803    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:02.919012 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:02.918986    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:32:03.136833 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:03.136806    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" path="/var/lib/kubelet/pods/e6c057fd-1612-4819-9f80-d40144c8c144/volumes"
Jul 13 20:32:03.921588 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:03.921533    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:32:07.922930 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:07.922899    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:32:07.923529 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:07.923500    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:08.926457 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:08.926430    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:32:08.927118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:08.927090    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:32:17.923908 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:17.923865    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:18.928045 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:18.928000    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:32:27.923892 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:27.923845    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:28.927185 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:28.927145    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:32:37.923743 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:37.923699    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:38.927319 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:38.927282    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:32:47.923464 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:47.923423    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:48.927288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:48.927249    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:32:54.701513 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:54.701477    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:32:57.923410 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:57.923371    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:32:58.927480 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:32:58.927437    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:33:07.924008 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:07.923969    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:33:08.928578 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:08.928543    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:33:11.133039 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:11.132997    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 13 20:33:11.298192 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:11.298152    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:21.135770 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:21.135744    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:33:36.223913 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:36.223870    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:36.753772 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:36.753739    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:46.296834 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.296797    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:46.459521 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.459490    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"]
Jul 13 20:33:46.459786 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.459745    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" containerID="cri-o://d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f" gracePeriod=30
Jul 13 20:33:46.459878 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.459820    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kube-rbac-proxy" containerID="cri-o://dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f" gracePeriod=30
Jul 13 20:33:46.463147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.463117    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:46.496744 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.496718    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"]
Jul 13 20:33:46.496975 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.496964    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container"
Jul 13 20:33:46.497020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.496977    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container"
Jul 13 20:33:46.497020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.496988    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kube-rbac-proxy"
Jul 13 20:33:46.497020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.496993    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kube-rbac-proxy"
Jul 13 20:33:46.497020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497001    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container"
Jul 13 20:33:46.497020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497007    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container"
Jul 13 20:33:46.497020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497018    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="storage-initializer"
Jul 13 20:33:46.497020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497023    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="storage-initializer"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497032    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="storage-initializer"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497038    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="storage-initializer"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497046    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kube-rbac-proxy"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497051    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kube-rbac-proxy"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497090    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kube-rbac-proxy"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497097    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kserve-container"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497105    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c057fd-1612-4819-9f80-d40144c8c144" containerName="kserve-container"
Jul 13 20:33:46.497221 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.497112    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="12d99135-2d02-439a-a77b-19064b7f0244" containerName="kube-rbac-proxy"
Jul 13 20:33:46.499812 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.499796    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.502246 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.502225    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-7d460-predictor-serving-cert\""
Jul 13 20:33:46.502420 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.502402    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-7d460-kube-rbac-proxy-sar-config\""
Jul 13 20:33:46.511253 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.511231    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"]
Jul 13 20:33:46.537001 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.536968    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:46.575097 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.575019    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"]
Jul 13 20:33:46.575356 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.575328    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" containerID="cri-o://c7915b2279e45c5c9c674582c59c6690dad5c16af258daf9110d5c778b340cb5" gracePeriod=30
Jul 13 20:33:46.575454 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.575367    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kube-rbac-proxy" containerID="cri-o://9c429734086454a163428dca2a8a2f1d2a5272f76449e6daab90f78243684391" gracePeriod=30
Jul 13 20:33:46.583016 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.582991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9327fd3-de24-42c1-af37-e78c88ac2fcd-proxy-tls\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.583114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.583044    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.583114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.583088    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnf2\" (UniqueName: \"kubernetes.io/projected/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-api-access-gnnf2\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.619961 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.619934    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:46.683764 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.683710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.683988 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.683793    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnf2\" (UniqueName: \"kubernetes.io/projected/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-api-access-gnnf2\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.683988 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.683888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9327fd3-de24-42c1-af37-e78c88ac2fcd-proxy-tls\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.684463 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.684438    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.686211 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.686187    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9327fd3-de24-42c1-af37-e78c88ac2fcd-proxy-tls\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.691776 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.691753    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnf2\" (UniqueName: \"kubernetes.io/projected/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-api-access-gnnf2\") pod \"message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") " pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.703113 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.703089    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:33:46.809344 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.809310    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:46.936760 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.936731    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"]
Jul 13 20:33:46.939845 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:33:46.939819    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9327fd3_de24_42c1_af37_e78c88ac2fcd.slice/crio-ac6a5f166d59ff78212139b1c25a4ea117e1c12f7a515995d37e7b1cf67be25b WatchSource:0}: Error finding container ac6a5f166d59ff78212139b1c25a4ea117e1c12f7a515995d37e7b1cf67be25b: Status 404 returned error can't find the container with id ac6a5f166d59ff78212139b1c25a4ea117e1c12f7a515995d37e7b1cf67be25b
Jul 13 20:33:46.941638 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:46.941621    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:33:47.291254 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:47.291175    2551 generic.go:356] "Generic (PLEG): container finished" podID="037c5f99-8d01-4345-97e9-01c25359948f" containerID="dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f" exitCode=2
Jul 13 20:33:47.291419 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:47.291248    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerDied","Data":"dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f"}
Jul 13 20:33:47.293172 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:47.293149    2551 generic.go:356] "Generic (PLEG): container finished" podID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerID="9c429734086454a163428dca2a8a2f1d2a5272f76449e6daab90f78243684391" exitCode=2
Jul 13 20:33:47.293314 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:47.293216    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerDied","Data":"9c429734086454a163428dca2a8a2f1d2a5272f76449e6daab90f78243684391"}
Jul 13 20:33:47.294208 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:47.294189    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" event={"ID":"b9327fd3-de24-42c1-af37-e78c88ac2fcd","Type":"ContainerStarted","Data":"ac6a5f166d59ff78212139b1c25a4ea117e1c12f7a515995d37e7b1cf67be25b"}
Jul 13 20:33:47.918646 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:47.918604    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 13 20:33:48.922084 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:48.922041    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 13 20:33:48.927419 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:48.927380    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 13 20:33:49.306093 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:49.306012    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" event={"ID":"b9327fd3-de24-42c1-af37-e78c88ac2fcd","Type":"ContainerStarted","Data":"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"}
Jul 13 20:33:49.306093 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:49.306072    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" event={"ID":"b9327fd3-de24-42c1-af37-e78c88ac2fcd","Type":"ContainerStarted","Data":"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"}
Jul 13 20:33:49.306296 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:49.306157    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:49.325191 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:49.325130    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" podStartSLOduration=1.966297472 podStartE2EDuration="3.325109795s" totalImagesPullingTime="1.358812323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:33:49.324136593 +0000 UTC m=+886.796365367" watchObservedRunningTime="2026-07-13 20:33:49.325109795 +0000 UTC m=+886.797338571"
Jul 13 20:33:50.318683 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.318095    2551 generic.go:356] "Generic (PLEG): container finished" podID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerID="c7915b2279e45c5c9c674582c59c6690dad5c16af258daf9110d5c778b340cb5" exitCode=0
Jul 13 20:33:50.318683 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.318188    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerDied","Data":"c7915b2279e45c5c9c674582c59c6690dad5c16af258daf9110d5c778b340cb5"}
Jul 13 20:33:50.319252 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.319014    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:50.320682 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.320639    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:33:50.415472 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.415450    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:33:50.511916 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.511827    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxmlc\" (UniqueName: \"kubernetes.io/projected/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-api-access-gxmlc\") pod \"6a595e79-b7c7-48ea-8280-abe131fa3159\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") "
Jul 13 20:33:50.511916 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.511868    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a595e79-b7c7-48ea-8280-abe131fa3159-proxy-tls\") pod \"6a595e79-b7c7-48ea-8280-abe131fa3159\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") "
Jul 13 20:33:50.511916 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.511911    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a595e79-b7c7-48ea-8280-abe131fa3159-kserve-provision-location\") pod \"6a595e79-b7c7-48ea-8280-abe131fa3159\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") "
Jul 13 20:33:50.512184 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.511933    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-cabundle-cert\") pod \"6a595e79-b7c7-48ea-8280-abe131fa3159\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") "
Jul 13 20:33:50.512184 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.511960    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-rbac-proxy-sar-config\") pod \"6a595e79-b7c7-48ea-8280-abe131fa3159\" (UID: \"6a595e79-b7c7-48ea-8280-abe131fa3159\") "
Jul 13 20:33:50.512288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.512266    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a595e79-b7c7-48ea-8280-abe131fa3159-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a595e79-b7c7-48ea-8280-abe131fa3159" (UID: "6a595e79-b7c7-48ea-8280-abe131fa3159"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:33:50.512288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.512279    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a595e79-b7c7-48ea-8280-abe131fa3159" (UID: "6a595e79-b7c7-48ea-8280-abe131fa3159"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:33:50.512415 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.512386    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a595e79-b7c7-48ea-8280-abe131fa3159" (UID: "6a595e79-b7c7-48ea-8280-abe131fa3159"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:33:50.513974 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.513952    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a595e79-b7c7-48ea-8280-abe131fa3159-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a595e79-b7c7-48ea-8280-abe131fa3159" (UID: "6a595e79-b7c7-48ea-8280-abe131fa3159"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:33:50.514106 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.513977    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-api-access-gxmlc" (OuterVolumeSpecName: "kube-api-access-gxmlc") pod "6a595e79-b7c7-48ea-8280-abe131fa3159" (UID: "6a595e79-b7c7-48ea-8280-abe131fa3159"). InnerVolumeSpecName "kube-api-access-gxmlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:33:50.612828 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.612796    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxmlc\" (UniqueName: \"kubernetes.io/projected/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-api-access-gxmlc\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:50.612828 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.612824    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a595e79-b7c7-48ea-8280-abe131fa3159-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:50.613024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.612840    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a595e79-b7c7-48ea-8280-abe131fa3159-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:50.613024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.612851    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:50.613024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:50.612860    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a595e79-b7c7-48ea-8280-abe131fa3159-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:51.183702 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.183673    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:33:51.216814 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.216788    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xc2\" (UniqueName: \"kubernetes.io/projected/037c5f99-8d01-4345-97e9-01c25359948f-kube-api-access-m8xc2\") pod \"037c5f99-8d01-4345-97e9-01c25359948f\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") "
Jul 13 20:33:51.216959 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.216829    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-cabundle-cert\") pod \"037c5f99-8d01-4345-97e9-01c25359948f\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") "
Jul 13 20:33:51.216959 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.216846    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037c5f99-8d01-4345-97e9-01c25359948f-proxy-tls\") pod \"037c5f99-8d01-4345-97e9-01c25359948f\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") "
Jul 13 20:33:51.216959 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.216868    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037c5f99-8d01-4345-97e9-01c25359948f-kserve-provision-location\") pod \"037c5f99-8d01-4345-97e9-01c25359948f\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") "
Jul 13 20:33:51.216959 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.216897    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-kube-rbac-proxy-sar-config\") pod \"037c5f99-8d01-4345-97e9-01c25359948f\" (UID: \"037c5f99-8d01-4345-97e9-01c25359948f\") "
Jul 13 20:33:51.217202 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.217174    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "037c5f99-8d01-4345-97e9-01c25359948f" (UID: "037c5f99-8d01-4345-97e9-01c25359948f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:33:51.217262 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.217190    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037c5f99-8d01-4345-97e9-01c25359948f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "037c5f99-8d01-4345-97e9-01c25359948f" (UID: "037c5f99-8d01-4345-97e9-01c25359948f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:33:51.217310 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.217262    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "037c5f99-8d01-4345-97e9-01c25359948f" (UID: "037c5f99-8d01-4345-97e9-01c25359948f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:33:51.218843 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.218821    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037c5f99-8d01-4345-97e9-01c25359948f-kube-api-access-m8xc2" (OuterVolumeSpecName: "kube-api-access-m8xc2") pod "037c5f99-8d01-4345-97e9-01c25359948f" (UID: "037c5f99-8d01-4345-97e9-01c25359948f"). InnerVolumeSpecName "kube-api-access-m8xc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:33:51.218843 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.218832    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037c5f99-8d01-4345-97e9-01c25359948f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "037c5f99-8d01-4345-97e9-01c25359948f" (UID: "037c5f99-8d01-4345-97e9-01c25359948f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:33:51.318056 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.317984    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:51.318056 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.318011    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xc2\" (UniqueName: \"kubernetes.io/projected/037c5f99-8d01-4345-97e9-01c25359948f-kube-api-access-m8xc2\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:51.318056 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.318021    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/037c5f99-8d01-4345-97e9-01c25359948f-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:51.318056 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.318029    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/037c5f99-8d01-4345-97e9-01c25359948f-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:51.318056 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.318038    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/037c5f99-8d01-4345-97e9-01c25359948f-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:33:51.323804 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.323780    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"
Jul 13 20:33:51.324176 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.323773    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk" event={"ID":"6a595e79-b7c7-48ea-8280-abe131fa3159","Type":"ContainerDied","Data":"d2caab08206007f073fc9dd03c15695f492a28fda457eb0c0c8b678d16fb618e"}
Jul 13 20:33:51.324176 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.323929    2551 scope.go:117] "RemoveContainer" containerID="9c429734086454a163428dca2a8a2f1d2a5272f76449e6daab90f78243684391"
Jul 13 20:33:51.325849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.325792    2551 generic.go:356] "Generic (PLEG): container finished" podID="037c5f99-8d01-4345-97e9-01c25359948f" containerID="d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f" exitCode=0
Jul 13 20:33:51.325849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.325821    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerDied","Data":"d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f"}
Jul 13 20:33:51.325849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.325842    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"
Jul 13 20:33:51.326144 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.325888    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" event={"ID":"037c5f99-8d01-4345-97e9-01c25359948f","Type":"ContainerDied","Data":"3507b85e1bda78938c3305c036d4543dc977ea806d3ffa7b17b9379c2121af29"}
Jul 13 20:33:51.332097 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.332081    2551 scope.go:117] "RemoveContainer" containerID="c7915b2279e45c5c9c674582c59c6690dad5c16af258daf9110d5c778b340cb5"
Jul 13 20:33:51.339152 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.339135    2551 scope.go:117] "RemoveContainer" containerID="39bb87df71f554c11b92bf042ea9a2b96f6ee4484da5c94d1cf64a16730b92f1"
Jul 13 20:33:51.340110 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.340092    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"]
Jul 13 20:33:51.346008 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.345971    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-553c9-predictor-58bb9f75cc-r2msk"]
Jul 13 20:33:51.347530 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.347514    2551 scope.go:117] "RemoveContainer" containerID="dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f"
Jul 13 20:33:51.354140 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.354125    2551 scope.go:117] "RemoveContainer" containerID="d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f"
Jul 13 20:33:51.361105 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.361089    2551 scope.go:117] "RemoveContainer" containerID="a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6"
Jul 13 20:33:51.361154 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.361108    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"]
Jul 13 20:33:51.365184 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.365166    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf"]
Jul 13 20:33:51.368553 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.368538    2551 scope.go:117] "RemoveContainer" containerID="dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f"
Jul 13 20:33:51.368838 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:33:51.368811    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f\": container with ID starting with dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f not found: ID does not exist" containerID="dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f"
Jul 13 20:33:51.368890 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.368848    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f"} err="failed to get container status \"dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f\": rpc error: code = NotFound desc = could not find container \"dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f\": container with ID starting with dff97cb32a933710c29bfe556a7ef2393c8688ff481312f30539e097b1d98f2f not found: ID does not exist"
Jul 13 20:33:51.368890 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.368865    2551 scope.go:117] "RemoveContainer" containerID="d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f"
Jul 13 20:33:51.369114 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:33:51.369097    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f\": container with ID starting with d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f not found: ID does not exist" containerID="d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f"
Jul 13 20:33:51.369170 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.369122    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f"} err="failed to get container status \"d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f\": rpc error: code = NotFound desc = could not find container \"d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f\": container with ID starting with d411adab8435c95aa8854e3635de09bccf68dccd8fd0ffceb8c924613c19ae5f not found: ID does not exist"
Jul 13 20:33:51.369170 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.369146    2551 scope.go:117] "RemoveContainer" containerID="a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6"
Jul 13 20:33:51.369394 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:33:51.369378    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6\": container with ID starting with a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6 not found: ID does not exist" containerID="a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6"
Jul 13 20:33:51.369428 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:51.369397    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6"} err="failed to get container status \"a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6\": rpc error: code = NotFound desc = could not find container \"a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6\": container with ID starting with a58ea88518ba823cfc64f23a579bc876263de7a79101a93f99cdaf309f0336c6 not found: ID does not exist"
Jul 13 20:33:52.133783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:52.133733    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-553c9-predictor-6bffb6f686-l5fgf" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: i/o timeout"
Jul 13 20:33:53.136411 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:53.136382    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037c5f99-8d01-4345-97e9-01c25359948f" path="/var/lib/kubelet/pods/037c5f99-8d01-4345-97e9-01c25359948f/volumes"
Jul 13 20:33:53.136866 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:53.136853    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" path="/var/lib/kubelet/pods/6a595e79-b7c7-48ea-8280-abe131fa3159/volumes"
Jul 13 20:33:57.335728 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:33:57.335637    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:34:02.975098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:02.975070    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:34:02.975544 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:02.975282    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:34:06.515071 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.515026    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:34:06.552535 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552504    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"]
Jul 13 20:34:06.552895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552881    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kube-rbac-proxy"
Jul 13 20:34:06.552943 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552898    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kube-rbac-proxy"
Jul 13 20:34:06.552943 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552914    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kube-rbac-proxy"
Jul 13 20:34:06.552943 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552923    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kube-rbac-proxy"
Jul 13 20:34:06.552943 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552939    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="storage-initializer"
Jul 13 20:34:06.553060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552948    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="storage-initializer"
Jul 13 20:34:06.553060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552961    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="storage-initializer"
Jul 13 20:34:06.553060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552970    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="storage-initializer"
Jul 13 20:34:06.553060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552980    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container"
Jul 13 20:34:06.553060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.552988    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container"
Jul 13 20:34:06.553060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.553001    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container"
Jul 13 20:34:06.553060 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.553009    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container"
Jul 13 20:34:06.553245 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.553061    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kube-rbac-proxy"
Jul 13 20:34:06.553245 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.553075    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="037c5f99-8d01-4345-97e9-01c25359948f" containerName="kserve-container"
Jul 13 20:34:06.553245 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.553083    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kserve-container"
Jul 13 20:34:06.553245 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.553092    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a595e79-b7c7-48ea-8280-abe131fa3159" containerName="kube-rbac-proxy"
Jul 13 20:34:06.560362 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.560309    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.562305 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.562281    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:34:06.562560 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.562541    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-7d460-predictor-serving-cert\""
Jul 13 20:34:06.562978 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.562959    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-7d460-kube-rbac-proxy-sar-config\""
Jul 13 20:34:06.563091 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.563031    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 20:34:06.566033 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.566009    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"]
Jul 13 20:34:06.617727 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.617693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-cabundle-cert\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.617849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.617741    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwnn\" (UniqueName: \"kubernetes.io/projected/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-api-access-cnwnn\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.617849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.617765    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.617849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.617827    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.617974 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.617861    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bb80a8c-1375-4804-9f47-0a0684ebe378-kserve-provision-location\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.718631 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.718597    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwnn\" (UniqueName: \"kubernetes.io/projected/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-api-access-cnwnn\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.718631 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.718633    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.718869 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.718699    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.718869 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.718729    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bb80a8c-1375-4804-9f47-0a0684ebe378-kserve-provision-location\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.718869 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:34:06.718838    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-serving-cert: secret "isvc-logger-raw-7d460-predictor-serving-cert" not found
Jul 13 20:34:06.719012 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.718870    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-cabundle-cert\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.719012 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:34:06.718918    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls podName:8bb80a8c-1375-4804-9f47-0a0684ebe378 nodeName:}" failed. No retries permitted until 2026-07-13 20:34:07.218898349 +0000 UTC m=+904.691127109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls") pod "isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" (UID: "8bb80a8c-1375-4804-9f47-0a0684ebe378") : secret "isvc-logger-raw-7d460-predictor-serving-cert" not found
Jul 13 20:34:06.719116 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.719100    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bb80a8c-1375-4804-9f47-0a0684ebe378-kserve-provision-location\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.719503 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.719482    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-cabundle-cert\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.720001 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.719981    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:06.727051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:06.727030    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwnn\" (UniqueName: \"kubernetes.io/projected/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-api-access-cnwnn\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:07.222288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:07.222252    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:07.224745 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:07.224712    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls\") pod \"isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") " pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:07.237345 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:07.237318    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:34:07.472816 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:07.472729    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:07.598760 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:07.598565    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"]
Jul 13 20:34:07.601475 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:34:07.601437    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb80a8c_1375_4804_9f47_0a0684ebe378.slice/crio-4a3896953e5395d72fc7e8485d4dc8b38e9a64c42057466f662c5a5626087f0d WatchSource:0}: Error finding container 4a3896953e5395d72fc7e8485d4dc8b38e9a64c42057466f662c5a5626087f0d: Status 404 returned error can't find the container with id 4a3896953e5395d72fc7e8485d4dc8b38e9a64c42057466f662c5a5626087f0d
Jul 13 20:34:08.384078 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:08.384044    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerStarted","Data":"0fbe28cc0fd3445ac792371e2e95dda1a75933d8027d58457092c00b1246c580"}
Jul 13 20:34:08.384253 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:08.384105    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerStarted","Data":"4a3896953e5395d72fc7e8485d4dc8b38e9a64c42057466f662c5a5626087f0d"}
Jul 13 20:34:11.395897 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:11.395865    2551 generic.go:356] "Generic (PLEG): container finished" podID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerID="0fbe28cc0fd3445ac792371e2e95dda1a75933d8027d58457092c00b1246c580" exitCode=0
Jul 13 20:34:11.396293 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:11.395932    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerDied","Data":"0fbe28cc0fd3445ac792371e2e95dda1a75933d8027d58457092c00b1246c580"}
Jul 13 20:34:12.401913 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:12.401875    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerStarted","Data":"d9c4faecdb77b6c30249ab43a265b925c02c41cda133489b9cfcd7c5f95ba882"}
Jul 13 20:34:12.402279 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:12.401943    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerStarted","Data":"eaefac1bcaa4690c08ed5ba5cef1e31d964f257615f8ab8a77a00641fdc67ccb"}
Jul 13 20:34:12.402279 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:12.401968    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerStarted","Data":"32e62067e6b5c1e5fe02f87b9f7d3db573c229914aae63a3ed8a4a8137b92c89"}
Jul 13 20:34:12.402279 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:12.402157    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:12.427589 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:12.425758    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podStartSLOduration=2.7738032969999997 podStartE2EDuration="6.425740228s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.651936931s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:34:12.423748011 +0000 UTC m=+909.895976785" watchObservedRunningTime="2026-07-13 20:34:12.425740228 +0000 UTC m=+909.897969003"
Jul 13 20:34:13.405924 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:13.405893    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:13.406288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:13.405963    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:13.407264 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:13.407235    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:34:13.407891 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:13.407856    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:34:14.409666 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:14.409608    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:34:14.410157 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:14.410126    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:34:19.413746 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:19.413715    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:34:19.414356 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:19.414323    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:34:19.414574 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:19.414549    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:34:29.414725 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:29.414685    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:34:29.415185 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:29.415154    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:34:39.414547 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:39.414504    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:34:39.414978 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:39.414898    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:34:47.991213 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:47.991175    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:34:49.414322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:49.414283    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:34:49.414776 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:49.414518    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:34:59.414567 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:59.414524    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:34:59.415239 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:34:59.415049    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:35:09.415079 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:09.415038    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:35:09.415561 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:09.415428    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:35:19.414225 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:19.414179    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:35:19.414703 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:19.414675    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:35:22.132122 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:22.132085    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:35:22.132541 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:22.132416    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:35:32.132851 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:32.132820    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:35:32.133450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:32.133420    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:35:41.555428 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.555387    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq_b9327fd3-de24-42c1-af37-e78c88ac2fcd/kserve-container/0.log"
Jul 13 20:35:41.791005 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.790965    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"]
Jul 13 20:35:41.791323 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.791272    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" containerID="cri-o://32e62067e6b5c1e5fe02f87b9f7d3db573c229914aae63a3ed8a4a8137b92c89" gracePeriod=30
Jul 13 20:35:41.791492 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.791327    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy" containerID="cri-o://eaefac1bcaa4690c08ed5ba5cef1e31d964f257615f8ab8a77a00641fdc67ccb" gracePeriod=30
Jul 13 20:35:41.791492 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.791290    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" containerID="cri-o://d9c4faecdb77b6c30249ab43a265b925c02c41cda133489b9cfcd7c5f95ba882" gracePeriod=30
Jul 13 20:35:41.799923 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.799890    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:35:41.869928 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.869895    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"]
Jul 13 20:35:41.873581 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.873559    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:41.875471 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.875446    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:35:41.875746 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.875724    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d5bc9-predictor-serving-cert\""
Jul 13 20:35:41.875838 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.875754    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-d5bc9-kube-rbac-proxy-sar-config\""
Jul 13 20:35:41.885111 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.885087    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"]
Jul 13 20:35:41.923734 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.923701    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:41.923858 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.923744    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:41.923858 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.923772    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7ld6\" (UniqueName: \"kubernetes.io/projected/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-api-access-k7ld6\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:41.923858 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.923843    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:41.923990 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.923876    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:41.930876 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.930852    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"]
Jul 13 20:35:41.931076 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.931054    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kserve-container" containerID="cri-o://3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1" gracePeriod=30
Jul 13 20:35:41.931156 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.931073    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kube-rbac-proxy" containerID="cri-o://e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c" gracePeriod=30
Jul 13 20:35:41.945919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:41.945894    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:35:42.025158 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.025109    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.025328 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.025170    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ld6\" (UniqueName: \"kubernetes.io/projected/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-api-access-k7ld6\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.025328 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.025228    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.025328 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.025259    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.025328 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:35:42.025279    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-serving-cert: secret "isvc-sklearn-scale-raw-d5bc9-predictor-serving-cert" not found
Jul 13 20:35:42.025328 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.025313    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.025583 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:35:42.025352    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls podName:863d6d12-4f15-4d3a-8be1-01f7d8043d55 nodeName:}" failed. No retries permitted until 2026-07-13 20:35:42.525330879 +0000 UTC m=+999.997559634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls") pod "isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" (UID: "863d6d12-4f15-4d3a-8be1-01f7d8043d55") : secret "isvc-sklearn-scale-raw-d5bc9-predictor-serving-cert" not found
Jul 13 20:35:42.025747 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.025725    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.026055 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.026035    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-cabundle-cert\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.026098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.026084    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.034198 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.034167    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ld6\" (UniqueName: \"kubernetes.io/projected/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-api-access-k7ld6\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.130622 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.130548    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:35:42.132584 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.132561    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:35:42.134194 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.134169    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:35:42.168723 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.168702    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:35:42.226957 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.226924    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnnf2\" (UniqueName: \"kubernetes.io/projected/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-api-access-gnnf2\") pod \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") "
Jul 13 20:35:42.227114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.226984    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-rbac-proxy-sar-config\") pod \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") "
Jul 13 20:35:42.227114 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.227049    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9327fd3-de24-42c1-af37-e78c88ac2fcd-proxy-tls\") pod \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\" (UID: \"b9327fd3-de24-42c1-af37-e78c88ac2fcd\") "
Jul 13 20:35:42.227378 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.227351    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9327fd3-de24-42c1-af37-e78c88ac2fcd" (UID: "b9327fd3-de24-42c1-af37-e78c88ac2fcd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:35:42.229032 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.229011    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9327fd3-de24-42c1-af37-e78c88ac2fcd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9327fd3-de24-42c1-af37-e78c88ac2fcd" (UID: "b9327fd3-de24-42c1-af37-e78c88ac2fcd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:35:42.229032 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.229020    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-api-access-gnnf2" (OuterVolumeSpecName: "kube-api-access-gnnf2") pod "b9327fd3-de24-42c1-af37-e78c88ac2fcd" (UID: "b9327fd3-de24-42c1-af37-e78c88ac2fcd"). InnerVolumeSpecName "kube-api-access-gnnf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:35:42.327684 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.327642    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9327fd3-de24-42c1-af37-e78c88ac2fcd-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:35:42.327684 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.327682    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnnf2\" (UniqueName: \"kubernetes.io/projected/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-api-access-gnnf2\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:35:42.327840 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.327693    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9327fd3-de24-42c1-af37-e78c88ac2fcd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:35:42.529309 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.529214    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.531590 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.531571    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls\") pod \"isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.717990 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.717955    2551 generic.go:356] "Generic (PLEG): container finished" podID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerID="eaefac1bcaa4690c08ed5ba5cef1e31d964f257615f8ab8a77a00641fdc67ccb" exitCode=2
Jul 13 20:35:42.718407 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.718037    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerDied","Data":"eaefac1bcaa4690c08ed5ba5cef1e31d964f257615f8ab8a77a00641fdc67ccb"}
Jul 13 20:35:42.719492 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.719468    2551 generic.go:356] "Generic (PLEG): container finished" podID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerID="e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c" exitCode=2
Jul 13 20:35:42.719492 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.719489    2551 generic.go:356] "Generic (PLEG): container finished" podID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerID="3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1" exitCode=2
Jul 13 20:35:42.719681 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.719514    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" event={"ID":"b9327fd3-de24-42c1-af37-e78c88ac2fcd","Type":"ContainerDied","Data":"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"}
Jul 13 20:35:42.719681 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.719536    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"
Jul 13 20:35:42.719681 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.719566    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" event={"ID":"b9327fd3-de24-42c1-af37-e78c88ac2fcd","Type":"ContainerDied","Data":"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"}
Jul 13 20:35:42.719681 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.719597    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq" event={"ID":"b9327fd3-de24-42c1-af37-e78c88ac2fcd","Type":"ContainerDied","Data":"ac6a5f166d59ff78212139b1c25a4ea117e1c12f7a515995d37e7b1cf67be25b"}
Jul 13 20:35:42.719879 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.719642    2551 scope.go:117] "RemoveContainer" containerID="e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"
Jul 13 20:35:42.727949 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.727932    2551 scope.go:117] "RemoveContainer" containerID="3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"
Jul 13 20:35:42.735157 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.735138    2551 scope.go:117] "RemoveContainer" containerID="e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"
Jul 13 20:35:42.735408 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:35:42.735389    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c\": container with ID starting with e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c not found: ID does not exist" containerID="e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"
Jul 13 20:35:42.735459 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.735418    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"} err="failed to get container status \"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c\": rpc error: code = NotFound desc = could not find container \"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c\": container with ID starting with e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c not found: ID does not exist"
Jul 13 20:35:42.735459 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.735436    2551 scope.go:117] "RemoveContainer" containerID="3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"
Jul 13 20:35:42.735714 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:35:42.735646    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1\": container with ID starting with 3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1 not found: ID does not exist" containerID="3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"
Jul 13 20:35:42.735764 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.735722    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"} err="failed to get container status \"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1\": rpc error: code = NotFound desc = could not find container \"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1\": container with ID starting with 3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1 not found: ID does not exist"
Jul 13 20:35:42.735764 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.735738    2551 scope.go:117] "RemoveContainer" containerID="e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"
Jul 13 20:35:42.735930 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.735913    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c"} err="failed to get container status \"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c\": rpc error: code = NotFound desc = could not find container \"e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c\": container with ID starting with e6681e554d6ec67b3dfeb9f3a897f976542934bffc553c2d093d9e513364929c not found: ID does not exist"
Jul 13 20:35:42.735971 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.735930    2551 scope.go:117] "RemoveContainer" containerID="3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"
Jul 13 20:35:42.736163 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.736144    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1"} err="failed to get container status \"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1\": rpc error: code = NotFound desc = could not find container \"3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1\": container with ID starting with 3925bfcff7c3e9f5dc1d2c2d889142c211c04bd93419082e0f58661c17f908a1 not found: ID does not exist"
Jul 13 20:35:42.739544 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.739521    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"]
Jul 13 20:35:42.745783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.745762    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-7d460-predictor-6fd47db4ff-v8lsq"]
Jul 13 20:35:42.786042 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.785968    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:42.918389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:42.918367    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"]
Jul 13 20:35:42.921138 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:35:42.921114    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863d6d12_4f15_4d3a_8be1_01f7d8043d55.slice/crio-437b94b6a6d3d0135cec5fdc2706855652b39aac6143667f19a8d50d620ed810 WatchSource:0}: Error finding container 437b94b6a6d3d0135cec5fdc2706855652b39aac6143667f19a8d50d620ed810: Status 404 returned error can't find the container with id 437b94b6a6d3d0135cec5fdc2706855652b39aac6143667f19a8d50d620ed810
Jul 13 20:35:43.136644 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:43.136612    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" path="/var/lib/kubelet/pods/b9327fd3-de24-42c1-af37-e78c88ac2fcd/volumes"
Jul 13 20:35:43.724883 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:43.724843    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerStarted","Data":"68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468"}
Jul 13 20:35:43.725371 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:43.724912    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerStarted","Data":"437b94b6a6d3d0135cec5fdc2706855652b39aac6143667f19a8d50d620ed810"}
Jul 13 20:35:44.410117 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:44.410070    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 13 20:35:46.738720 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:46.738694    2551 generic.go:356] "Generic (PLEG): container finished" podID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerID="32e62067e6b5c1e5fe02f87b9f7d3db573c229914aae63a3ed8a4a8137b92c89" exitCode=0
Jul 13 20:35:46.739097 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:46.738764    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerDied","Data":"32e62067e6b5c1e5fe02f87b9f7d3db573c229914aae63a3ed8a4a8137b92c89"}
Jul 13 20:35:46.740107 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:46.740089    2551 generic.go:356] "Generic (PLEG): container finished" podID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerID="68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468" exitCode=0
Jul 13 20:35:46.740170 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:46.740140    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerDied","Data":"68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468"}
Jul 13 20:35:47.745252 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:47.745217    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerStarted","Data":"a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f"}
Jul 13 20:35:47.745616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:47.745309    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerStarted","Data":"2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1"}
Jul 13 20:35:47.745616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:47.745499    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:47.765510 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:47.765460    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podStartSLOduration=3.029495012 podStartE2EDuration="6.765445077s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.735950065s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:35:47.764408434 +0000 UTC m=+1005.236637208" watchObservedRunningTime="2026-07-13 20:35:47.765445077 +0000 UTC m=+1005.237673850"
Jul 13 20:35:48.748996 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:48.748956    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:48.750096 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:48.750073    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:35:49.410644 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:49.410599    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 13 20:35:49.752494 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:49.752405    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:35:52.132198 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:52.132159    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:35:52.133816 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:52.133789    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:35:54.410830 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:54.410781    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 13 20:35:54.411202 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:54.410921    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:35:54.758210 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:54.758136    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:35:54.758692 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:54.758644    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:35:59.410416 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:35:59.410376    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 13 20:36:02.132293 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:02.132247    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:36:02.132638 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:02.132387    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:36:02.133890 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:02.133857    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 13 20:36:02.133995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:02.133981    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:36:04.410619 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:04.410574    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 13 20:36:04.759178 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:04.759097    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:36:09.409909 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:09.409857    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 13 20:36:11.834359 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:11.834333    2551 generic.go:356] "Generic (PLEG): container finished" podID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerID="d9c4faecdb77b6c30249ab43a265b925c02c41cda133489b9cfcd7c5f95ba882" exitCode=0
Jul 13 20:36:11.834703 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:11.834402    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerDied","Data":"d9c4faecdb77b6c30249ab43a265b925c02c41cda133489b9cfcd7c5f95ba882"}
Jul 13 20:36:12.132006 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.131973    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 13 20:36:12.134338 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.134308    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent" probeResult="failure" output="Get \"http://10.133.0.23:9081/\": dial tcp 10.133.0.23:9081: connect: connection refused"
Jul 13 20:36:12.435474 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.435450    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:36:12.558163 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558130    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-cabundle-cert\") pod \"8bb80a8c-1375-4804-9f47-0a0684ebe378\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") "
Jul 13 20:36:12.558163 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558167    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnwnn\" (UniqueName: \"kubernetes.io/projected/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-api-access-cnwnn\") pod \"8bb80a8c-1375-4804-9f47-0a0684ebe378\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") "
Jul 13 20:36:12.558409 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558189    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-rbac-proxy-sar-config\") pod \"8bb80a8c-1375-4804-9f47-0a0684ebe378\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") "
Jul 13 20:36:12.558409 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558212    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bb80a8c-1375-4804-9f47-0a0684ebe378-kserve-provision-location\") pod \"8bb80a8c-1375-4804-9f47-0a0684ebe378\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") "
Jul 13 20:36:12.558409 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558240    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls\") pod \"8bb80a8c-1375-4804-9f47-0a0684ebe378\" (UID: \"8bb80a8c-1375-4804-9f47-0a0684ebe378\") "
Jul 13 20:36:12.558567 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558486    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb80a8c-1375-4804-9f47-0a0684ebe378-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bb80a8c-1375-4804-9f47-0a0684ebe378" (UID: "8bb80a8c-1375-4804-9f47-0a0684ebe378"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:36:12.558567 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558545    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bb80a8c-1375-4804-9f47-0a0684ebe378" (UID: "8bb80a8c-1375-4804-9f47-0a0684ebe378"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:36:12.558646 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.558558    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bb80a8c-1375-4804-9f47-0a0684ebe378" (UID: "8bb80a8c-1375-4804-9f47-0a0684ebe378"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:36:12.560275 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.560253    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bb80a8c-1375-4804-9f47-0a0684ebe378" (UID: "8bb80a8c-1375-4804-9f47-0a0684ebe378"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:36:12.560406 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.560385    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-api-access-cnwnn" (OuterVolumeSpecName: "kube-api-access-cnwnn") pod "8bb80a8c-1375-4804-9f47-0a0684ebe378" (UID: "8bb80a8c-1375-4804-9f47-0a0684ebe378"). InnerVolumeSpecName "kube-api-access-cnwnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:36:12.659164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.659126    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bb80a8c-1375-4804-9f47-0a0684ebe378-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:36:12.659164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.659158    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:36:12.659164 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.659168    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwnn\" (UniqueName: \"kubernetes.io/projected/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-api-access-cnwnn\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:36:12.659386 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.659179    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bb80a8c-1375-4804-9f47-0a0684ebe378-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:36:12.659386 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.659188    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bb80a8c-1375-4804-9f47-0a0684ebe378-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:36:12.840302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.840221    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5" event={"ID":"8bb80a8c-1375-4804-9f47-0a0684ebe378","Type":"ContainerDied","Data":"4a3896953e5395d72fc7e8485d4dc8b38e9a64c42057466f662c5a5626087f0d"}
Jul 13 20:36:12.840302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.840245    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"
Jul 13 20:36:12.840302 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.840293    2551 scope.go:117] "RemoveContainer" containerID="d9c4faecdb77b6c30249ab43a265b925c02c41cda133489b9cfcd7c5f95ba882"
Jul 13 20:36:12.848618 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.848599    2551 scope.go:117] "RemoveContainer" containerID="eaefac1bcaa4690c08ed5ba5cef1e31d964f257615f8ab8a77a00641fdc67ccb"
Jul 13 20:36:12.855549 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.855531    2551 scope.go:117] "RemoveContainer" containerID="32e62067e6b5c1e5fe02f87b9f7d3db573c229914aae63a3ed8a4a8137b92c89"
Jul 13 20:36:12.862196 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.862175    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"]
Jul 13 20:36:12.862364 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.862351    2551 scope.go:117] "RemoveContainer" containerID="0fbe28cc0fd3445ac792371e2e95dda1a75933d8027d58457092c00b1246c580"
Jul 13 20:36:12.866153 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:12.866133    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-7d460-predictor-857dd785fb-4mkx5"]
Jul 13 20:36:13.139533 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:13.139503    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" path="/var/lib/kubelet/pods/8bb80a8c-1375-4804-9f47-0a0684ebe378/volumes"
Jul 13 20:36:14.759562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:14.759527    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:36:24.758874 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:24.758837    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:36:34.759605 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:34.759566    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:36:44.759170 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:44.759132    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:36:54.759562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:36:54.759475    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:37:04.132168 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:37:04.132118    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:37:14.132111 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:37:14.132063    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:37:24.132621 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:37:24.132584    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:37:34.132022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:37:34.131977    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:37:44.132781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:37:44.132740    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:37:54.132009 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:37:54.131969    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:38:04.132161 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:04.132117    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:38:14.132781 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:14.132739    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:38:18.131929 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:18.131885    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:38:26.623395 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:26.623319    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:38:28.132569 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:28.132537    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:38:32.129585 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:32.129553    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"]
Jul 13 20:38:32.130014 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:32.129810    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" containerID="cri-o://2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1" gracePeriod=30
Jul 13 20:38:32.130014 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:32.129843    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kube-rbac-proxy" containerID="cri-o://a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f" gracePeriod=30
Jul 13 20:38:32.139497 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:32.139472    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:38:32.329397 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:32.329368    2551 generic.go:356] "Generic (PLEG): container finished" podID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerID="a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f" exitCode=2
Jul 13 20:38:32.329550 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:32.329412    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerDied","Data":"a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f"}
Jul 13 20:38:34.753055 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:34.753011    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 13 20:38:37.198100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.198068    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:38:37.299527 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299491    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"]
Jul 13 20:38:37.299775 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299763    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kserve-container"
Jul 13 20:38:37.299822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299777    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kserve-container"
Jul 13 20:38:37.299822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299786    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="storage-initializer"
Jul 13 20:38:37.299822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299791    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="storage-initializer"
Jul 13 20:38:37.299822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299797    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy"
Jul 13 20:38:37.299822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299803    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy"
Jul 13 20:38:37.299822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299817    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container"
Jul 13 20:38:37.299822 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299822    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299832    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299837    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299848    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kube-rbac-proxy"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299854    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kube-rbac-proxy"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299892    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kube-rbac-proxy"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299898    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kube-rbac-proxy"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299905    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="kserve-container"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299911    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9327fd3-de24-42c1-af37-e78c88ac2fcd" containerName="kserve-container"
Jul 13 20:38:37.300022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.299916    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb80a8c-1375-4804-9f47-0a0684ebe378" containerName="agent"
Jul 13 20:38:37.302800 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.302783    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.305747 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.305727    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-fa5c0a-predictor-serving-cert\""
Jul 13 20:38:37.306137 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.306120    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-fa5c0a-kube-rbac-proxy-sar-config\""
Jul 13 20:38:37.309350 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.309331    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"]
Jul 13 20:38:37.355816 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.355785    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:38:37.407249 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.407224    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e576447-f7a9-4f98-bcf9-fd287d21acd7-proxy-tls\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.407389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.407268    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.407389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.407295    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9jd\" (UniqueName: \"kubernetes.io/projected/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-api-access-fk9jd\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.407389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.407348    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kserve-provision-location\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.407389 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.407380    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-cabundle-cert\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.508625 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.508525    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e576447-f7a9-4f98-bcf9-fd287d21acd7-proxy-tls\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.508625 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.508585    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.508625 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.508623    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9jd\" (UniqueName: \"kubernetes.io/projected/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-api-access-fk9jd\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.508941 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.508639    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kserve-provision-location\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.508941 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.508677    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-cabundle-cert\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.509176 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.509153    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kserve-provision-location\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.509383 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.509362    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-cabundle-cert\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.509458 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.509428    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-rbac-proxy-sar-config\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.511083 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.511063    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e576447-f7a9-4f98-bcf9-fd287d21acd7-proxy-tls\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.518136 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.518106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9jd\" (UniqueName: \"kubernetes.io/projected/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-api-access-fk9jd\") pod \"isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") " pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.613792 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.613756    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:37.746309 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:37.746277    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"]
Jul 13 20:38:37.749983 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:38:37.749953    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e576447_f7a9_4f98_bcf9_fd287d21acd7.slice/crio-ed5195f3205d6b4ac5995e7a6ee14680230550583c00e6c368e5663cd1d31f4b WatchSource:0}: Error finding container ed5195f3205d6b4ac5995e7a6ee14680230550583c00e6c368e5663cd1d31f4b: Status 404 returned error can't find the container with id ed5195f3205d6b4ac5995e7a6ee14680230550583c00e6c368e5663cd1d31f4b
Jul 13 20:38:38.132246 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:38.132197    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 13 20:38:38.350366 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:38.350332    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerStarted","Data":"e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb"}
Jul 13 20:38:38.350778 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:38.350394    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerStarted","Data":"ed5195f3205d6b4ac5995e7a6ee14680230550583c00e6c368e5663cd1d31f4b"}
Jul 13 20:38:39.753456 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:39.753413    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 13 20:38:41.773927 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.773905    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:38:41.841005 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.840977    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-rbac-proxy-sar-config\") pod \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") "
Jul 13 20:38:41.841166 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.841030    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls\") pod \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") "
Jul 13 20:38:41.841166 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.841048    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7ld6\" (UniqueName: \"kubernetes.io/projected/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-api-access-k7ld6\") pod \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") "
Jul 13 20:38:41.841166 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.841065    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kserve-provision-location\") pod \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") "
Jul 13 20:38:41.841325 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.841174    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-cabundle-cert\") pod \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\" (UID: \"863d6d12-4f15-4d3a-8be1-01f7d8043d55\") "
Jul 13 20:38:41.841399 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.841373    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "863d6d12-4f15-4d3a-8be1-01f7d8043d55" (UID: "863d6d12-4f15-4d3a-8be1-01f7d8043d55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:38:41.841513 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.841493    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "863d6d12-4f15-4d3a-8be1-01f7d8043d55" (UID: "863d6d12-4f15-4d3a-8be1-01f7d8043d55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:38:41.841558 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.841540    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "863d6d12-4f15-4d3a-8be1-01f7d8043d55" (UID: "863d6d12-4f15-4d3a-8be1-01f7d8043d55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:38:41.843120 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.843098    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "863d6d12-4f15-4d3a-8be1-01f7d8043d55" (UID: "863d6d12-4f15-4d3a-8be1-01f7d8043d55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:38:41.843219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.843101    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-api-access-k7ld6" (OuterVolumeSpecName: "kube-api-access-k7ld6") pod "863d6d12-4f15-4d3a-8be1-01f7d8043d55" (UID: "863d6d12-4f15-4d3a-8be1-01f7d8043d55"). InnerVolumeSpecName "kube-api-access-k7ld6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:38:41.942536 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.942454    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/863d6d12-4f15-4d3a-8be1-01f7d8043d55-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:38:41.942536 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.942490    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7ld6\" (UniqueName: \"kubernetes.io/projected/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-api-access-k7ld6\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:38:41.942536 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.942500    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:38:41.942536 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.942508    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:38:41.942536 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:41.942517    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/863d6d12-4f15-4d3a-8be1-01f7d8043d55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:38:42.365579 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.365547    2551 generic.go:356] "Generic (PLEG): container finished" podID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerID="2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1" exitCode=0
Jul 13 20:38:42.365791 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.365612    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"
Jul 13 20:38:42.365791 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.365630    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerDied","Data":"2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1"}
Jul 13 20:38:42.365791 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.365715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4" event={"ID":"863d6d12-4f15-4d3a-8be1-01f7d8043d55","Type":"ContainerDied","Data":"437b94b6a6d3d0135cec5fdc2706855652b39aac6143667f19a8d50d620ed810"}
Jul 13 20:38:42.365791 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.365750    2551 scope.go:117] "RemoveContainer" containerID="a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f"
Jul 13 20:38:42.367159 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.367134    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerID="e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb" exitCode=0
Jul 13 20:38:42.367248 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.367174    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerDied","Data":"e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb"}
Jul 13 20:38:42.374376 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.374351    2551 scope.go:117] "RemoveContainer" containerID="2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1"
Jul 13 20:38:42.381643 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.381622    2551 scope.go:117] "RemoveContainer" containerID="68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468"
Jul 13 20:38:42.390024 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.390006    2551 scope.go:117] "RemoveContainer" containerID="a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f"
Jul 13 20:38:42.390262 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:38:42.390244    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f\": container with ID starting with a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f not found: ID does not exist" containerID="a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f"
Jul 13 20:38:42.390299 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.390271    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f"} err="failed to get container status \"a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f\": rpc error: code = NotFound desc = could not find container \"a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f\": container with ID starting with a43c2241f9f1ea4d8525f898f4ba282b123064fad22122d660da0d686159f58f not found: ID does not exist"
Jul 13 20:38:42.390299 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.390289    2551 scope.go:117] "RemoveContainer" containerID="2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1"
Jul 13 20:38:42.390518 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:38:42.390501    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1\": container with ID starting with 2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1 not found: ID does not exist" containerID="2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1"
Jul 13 20:38:42.390555 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.390524    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1"} err="failed to get container status \"2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1\": rpc error: code = NotFound desc = could not find container \"2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1\": container with ID starting with 2e1a4752618641ee9ba88825bc1362be5e1f4debd461f2c1c6ac35b53a49f0c1 not found: ID does not exist"
Jul 13 20:38:42.390555 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.390537    2551 scope.go:117] "RemoveContainer" containerID="68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468"
Jul 13 20:38:42.390819 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:38:42.390803    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468\": container with ID starting with 68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468 not found: ID does not exist" containerID="68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468"
Jul 13 20:38:42.390876 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.390826    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468"} err="failed to get container status \"68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468\": rpc error: code = NotFound desc = could not find container \"68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468\": container with ID starting with 68b6cf2b1c8caa0d5d38ed834109fafe5fc76fd24034c189020c896c0e89e468 not found: ID does not exist"
Jul 13 20:38:42.409298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.409271    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"]
Jul 13 20:38:42.414734 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:42.414712    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-d5bc9-predictor-697fb7b44d-n5fh4"]
Jul 13 20:38:43.136422 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:43.136391    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" path="/var/lib/kubelet/pods/863d6d12-4f15-4d3a-8be1-01f7d8043d55/volumes"
Jul 13 20:38:43.373488 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:43.373456    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerStarted","Data":"4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988"}
Jul 13 20:38:43.373646 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:43.373513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerStarted","Data":"91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238"}
Jul 13 20:38:43.373817 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:43.373791    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:43.373945 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:43.373928    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:43.375064 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:43.375037    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:38:43.400548 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:43.400464    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podStartSLOduration=2.588810484 podStartE2EDuration="6.400451304s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81164082s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:38:43.398173308 +0000 UTC m=+1180.870402094" watchObservedRunningTime="2026-07-13 20:38:43.400451304 +0000 UTC m=+1180.872680077"
Jul 13 20:38:44.378133 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:44.378094    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:38:49.384107 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:49.384076    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:38:49.384711 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:49.384684    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:38:59.385413 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:38:59.385377    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:39:02.994925 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:02.994899    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:39:02.995813 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:02.995793    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:39:09.385670 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:09.385610    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:39:19.385082 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:19.385036    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:39:29.385046 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:29.385005    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:39:39.384883 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:39.384843    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:39:49.385570 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:49.385531    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 13 20:39:57.136287 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:39:57.136216    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:40:07.489309 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.489275    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:07.534875 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.534842    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"]
Jul 13 20:40:07.535106 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535095    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kube-rbac-proxy"
Jul 13 20:40:07.535147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535108    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kube-rbac-proxy"
Jul 13 20:40:07.535147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535125    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="storage-initializer"
Jul 13 20:40:07.535147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535131    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="storage-initializer"
Jul 13 20:40:07.535147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535139    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container"
Jul 13 20:40:07.535147 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535145    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container"
Jul 13 20:40:07.535295 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535191    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kube-rbac-proxy"
Jul 13 20:40:07.535295 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.535199    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="863d6d12-4f15-4d3a-8be1-01f7d8043d55" containerName="kserve-container"
Jul 13 20:40:07.537874 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.537859    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.540695 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.540666    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-fa5c0a-dockercfg-crnjz\""
Jul 13 20:40:07.540823 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.540736    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-fa5c0a\""
Jul 13 20:40:07.541025 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.541010    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-fa5c0a-predictor-serving-cert\""
Jul 13 20:40:07.541088 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.541026    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-fa5c0a-kube-rbac-proxy-sar-config\""
Jul 13 20:40:07.550880 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.550859    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"]
Jul 13 20:40:07.569439 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.569415    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dns\" (UniqueName: \"kubernetes.io/projected/05fc1166-d332-419f-843e-043029fc065f-kube-api-access-m5dns\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.569556 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.569446    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.569556 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.569466    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-cabundle-cert\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.569556 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.569503    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05fc1166-d332-419f-843e-043029fc065f-kserve-provision-location\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.569556 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.569528    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05fc1166-d332-419f-843e-043029fc065f-proxy-tls\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.670216 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.670181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dns\" (UniqueName: \"kubernetes.io/projected/05fc1166-d332-419f-843e-043029fc065f-kube-api-access-m5dns\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.670540 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.670217    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.670540 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.670249    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-cabundle-cert\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.670540 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.670286    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05fc1166-d332-419f-843e-043029fc065f-kserve-provision-location\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.670540 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.670316    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05fc1166-d332-419f-843e-043029fc065f-proxy-tls\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.670895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.670775    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05fc1166-d332-419f-843e-043029fc065f-kserve-provision-location\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.670962 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.670932    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.671064 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.671045    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-cabundle-cert\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.673152 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.673127    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05fc1166-d332-419f-843e-043029fc065f-proxy-tls\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.680961 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.680925    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dns\" (UniqueName: \"kubernetes.io/projected/05fc1166-d332-419f-843e-043029fc065f-kube-api-access-m5dns\") pod \"isvc-secondary-fa5c0a-predictor-75cc68664-cg66b\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") " pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.847317 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.847276    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:07.991154 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:07.991120    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"]
Jul 13 20:40:07.994540 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:40:07.994512    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fc1166_d332_419f_843e_043029fc065f.slice/crio-eb091786c515754b67d4b881341eea1441ac4fece454a84de3871607a22b9a89 WatchSource:0}: Error finding container eb091786c515754b67d4b881341eea1441ac4fece454a84de3871607a22b9a89: Status 404 returned error can't find the container with id eb091786c515754b67d4b881341eea1441ac4fece454a84de3871607a22b9a89
Jul 13 20:40:08.667691 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:08.667640    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b" event={"ID":"05fc1166-d332-419f-843e-043029fc065f","Type":"ContainerStarted","Data":"5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a"}
Jul 13 20:40:08.668079 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:08.667711    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b" event={"ID":"05fc1166-d332-419f-843e-043029fc065f","Type":"ContainerStarted","Data":"eb091786c515754b67d4b881341eea1441ac4fece454a84de3871607a22b9a89"}
Jul 13 20:40:09.005921 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:09.005828    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:10.676878 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:10.676853    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_05fc1166-d332-419f-843e-043029fc065f/storage-initializer/0.log"
Jul 13 20:40:10.677246 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:10.676892    2551 generic.go:356] "Generic (PLEG): container finished" podID="05fc1166-d332-419f-843e-043029fc065f" containerID="5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a" exitCode=1
Jul 13 20:40:10.677246 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:10.676947    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b" event={"ID":"05fc1166-d332-419f-843e-043029fc065f","Type":"ContainerDied","Data":"5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a"}
Jul 13 20:40:11.682832 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:11.682805    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_05fc1166-d332-419f-843e-043029fc065f/storage-initializer/0.log"
Jul 13 20:40:11.683204 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:11.682900    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b" event={"ID":"05fc1166-d332-419f-843e-043029fc065f","Type":"ContainerStarted","Data":"9fad8ef428ee65c8a087f4d2374f151f125fbfe01c1be4bdd120e9c0d6f97184"}
Jul 13 20:40:15.699470 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:15.699441    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_05fc1166-d332-419f-843e-043029fc065f/storage-initializer/1.log"
Jul 13 20:40:15.699892 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:15.699811    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_05fc1166-d332-419f-843e-043029fc065f/storage-initializer/0.log"
Jul 13 20:40:15.699892 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:15.699839    2551 generic.go:356] "Generic (PLEG): container finished" podID="05fc1166-d332-419f-843e-043029fc065f" containerID="9fad8ef428ee65c8a087f4d2374f151f125fbfe01c1be4bdd120e9c0d6f97184" exitCode=1
Jul 13 20:40:15.699892 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:15.699875    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b" event={"ID":"05fc1166-d332-419f-843e-043029fc065f","Type":"ContainerDied","Data":"9fad8ef428ee65c8a087f4d2374f151f125fbfe01c1be4bdd120e9c0d6f97184"}
Jul 13 20:40:15.699990 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:15.699929    2551 scope.go:117] "RemoveContainer" containerID="5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a"
Jul 13 20:40:15.700269 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:15.700242    2551 scope.go:117] "RemoveContainer" containerID="5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a"
Jul 13 20:40:15.710519 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:15.710496    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_kserve-ci-e2e-test(05fc1166-d332-419f-843e-043029fc065f)\"" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b" podUID="05fc1166-d332-419f-843e-043029fc065f"
Jul 13 20:40:15.710716 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:15.710631    2551 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_kserve-ci-e2e-test_05fc1166-d332-419f-843e-043029fc065f_0 in pod sandbox eb091786c515754b67d4b881341eea1441ac4fece454a84de3871607a22b9a89 from index: no such id: '5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a'" containerID="5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a"
Jul 13 20:40:15.710716 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:15.710681    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_kserve-ci-e2e-test_05fc1166-d332-419f-843e-043029fc065f_0 in pod sandbox eb091786c515754b67d4b881341eea1441ac4fece454a84de3871607a22b9a89 from index: no such id: '5246dcac763864812a8d2681920004e65634f88e66f217562b893a163ea05e3a'"
Jul 13 20:40:16.704467 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:16.704437    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_05fc1166-d332-419f-843e-043029fc065f/storage-initializer/1.log"
Jul 13 20:40:21.593970 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.593933    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"]
Jul 13 20:40:21.609194 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.609168    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:21.687487 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.687453    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"]
Jul 13 20:40:21.687821 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.687763    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container" containerID="cri-o://91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238" gracePeriod=30
Jul 13 20:40:21.687942 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.687811    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kube-rbac-proxy" containerID="cri-o://4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988" gracePeriod=30
Jul 13 20:40:21.699855 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.699832    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:21.805193 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.805170    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_05fc1166-d332-419f-843e-043029fc065f/storage-initializer/1.log"
Jul 13 20:40:21.807214 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.807198    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:21.873514 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873437    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5dns\" (UniqueName: \"kubernetes.io/projected/05fc1166-d332-419f-843e-043029fc065f-kube-api-access-m5dns\") pod \"05fc1166-d332-419f-843e-043029fc065f\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") "
Jul 13 20:40:21.873514 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873479    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-kube-rbac-proxy-sar-config\") pod \"05fc1166-d332-419f-843e-043029fc065f\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") "
Jul 13 20:40:21.873514 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873512    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-cabundle-cert\") pod \"05fc1166-d332-419f-843e-043029fc065f\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") "
Jul 13 20:40:21.873765 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873529    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05fc1166-d332-419f-843e-043029fc065f-kserve-provision-location\") pod \"05fc1166-d332-419f-843e-043029fc065f\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") "
Jul 13 20:40:21.873765 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873567    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05fc1166-d332-419f-843e-043029fc065f-proxy-tls\") pod \"05fc1166-d332-419f-843e-043029fc065f\" (UID: \"05fc1166-d332-419f-843e-043029fc065f\") "
Jul 13 20:40:21.873904 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873871    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc1166-d332-419f-843e-043029fc065f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05fc1166-d332-419f-843e-043029fc065f" (UID: "05fc1166-d332-419f-843e-043029fc065f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:40:21.873968 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873946    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05fc1166-d332-419f-843e-043029fc065f" (UID: "05fc1166-d332-419f-843e-043029fc065f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:21.874005 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.873951    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05fc1166-d332-419f-843e-043029fc065f" (UID: "05fc1166-d332-419f-843e-043029fc065f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:21.875712 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.875679    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fc1166-d332-419f-843e-043029fc065f-kube-api-access-m5dns" (OuterVolumeSpecName: "kube-api-access-m5dns") pod "05fc1166-d332-419f-843e-043029fc065f" (UID: "05fc1166-d332-419f-843e-043029fc065f"). InnerVolumeSpecName "kube-api-access-m5dns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:21.875802 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.875677    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fc1166-d332-419f-843e-043029fc065f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05fc1166-d332-419f-843e-043029fc065f" (UID: "05fc1166-d332-419f-843e-043029fc065f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:21.974297 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.974262    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:21.974297 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.974294    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05fc1166-d332-419f-843e-043029fc065f-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:21.974297 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.974303    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05fc1166-d332-419f-843e-043029fc065f-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:21.974529 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.974313    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05fc1166-d332-419f-843e-043029fc065f-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:21.974529 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:21.974323    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5dns\" (UniqueName: \"kubernetes.io/projected/05fc1166-d332-419f-843e-043029fc065f-kube-api-access-m5dns\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:22.727174 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.727146    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-fa5c0a-predictor-75cc68664-cg66b_05fc1166-d332-419f-843e-043029fc065f/storage-initializer/1.log"
Jul 13 20:40:22.727612 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.727231    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b" event={"ID":"05fc1166-d332-419f-843e-043029fc065f","Type":"ContainerDied","Data":"eb091786c515754b67d4b881341eea1441ac4fece454a84de3871607a22b9a89"}
Jul 13 20:40:22.727612 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.727243    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"
Jul 13 20:40:22.727612 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.727288    2551 scope.go:117] "RemoveContainer" containerID="9fad8ef428ee65c8a087f4d2374f151f125fbfe01c1be4bdd120e9c0d6f97184"
Jul 13 20:40:22.729583 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.729558    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerID="4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988" exitCode=2
Jul 13 20:40:22.729777 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.729631    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerDied","Data":"4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988"}
Jul 13 20:40:22.771775 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.771736    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"]
Jul 13 20:40:22.780839 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:22.780816    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-fa5c0a-predictor-75cc68664-cg66b"]
Jul 13 20:40:23.136492 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:23.136460    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05fc1166-d332-419f-843e-043029fc065f" path="/var/lib/kubelet/pods/05fc1166-d332-419f-843e-043029fc065f/volumes"
Jul 13 20:40:24.378364 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:24.378322    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 13 20:40:26.429122 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.429097    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:40:26.507583 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.507490    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kserve-provision-location\") pod \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") "
Jul 13 20:40:26.507583 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.507545    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e576447-f7a9-4f98-bcf9-fd287d21acd7-proxy-tls\") pod \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") "
Jul 13 20:40:26.507583 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.507571    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk9jd\" (UniqueName: \"kubernetes.io/projected/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-api-access-fk9jd\") pod \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") "
Jul 13 20:40:26.507885 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.507609    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-cabundle-cert\") pod \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") "
Jul 13 20:40:26.507885 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.507642    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-rbac-proxy-sar-config\") pod \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\" (UID: \"2e576447-f7a9-4f98-bcf9-fd287d21acd7\") "
Jul 13 20:40:26.508028 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.507932    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e576447-f7a9-4f98-bcf9-fd287d21acd7" (UID: "2e576447-f7a9-4f98-bcf9-fd287d21acd7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:40:26.508100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.508005    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e576447-f7a9-4f98-bcf9-fd287d21acd7" (UID: "2e576447-f7a9-4f98-bcf9-fd287d21acd7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:26.508100 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.508074    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e576447-f7a9-4f98-bcf9-fd287d21acd7" (UID: "2e576447-f7a9-4f98-bcf9-fd287d21acd7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:26.509724 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.509700    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e576447-f7a9-4f98-bcf9-fd287d21acd7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e576447-f7a9-4f98-bcf9-fd287d21acd7" (UID: "2e576447-f7a9-4f98-bcf9-fd287d21acd7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:26.509849 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.509831    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-api-access-fk9jd" (OuterVolumeSpecName: "kube-api-access-fk9jd") pod "2e576447-f7a9-4f98-bcf9-fd287d21acd7" (UID: "2e576447-f7a9-4f98-bcf9-fd287d21acd7"). InnerVolumeSpecName "kube-api-access-fk9jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:26.609098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.609064    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:26.609098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.609093    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:26.609098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.609103    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:26.609316 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.609114    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e576447-f7a9-4f98-bcf9-fd287d21acd7-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:26.609316 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.609123    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9jd\" (UniqueName: \"kubernetes.io/projected/2e576447-f7a9-4f98-bcf9-fd287d21acd7-kube-api-access-fk9jd\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:26.746563 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.746529    2551 generic.go:356] "Generic (PLEG): container finished" podID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerID="91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238" exitCode=0
Jul 13 20:40:26.746774 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.746586    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"
Jul 13 20:40:26.746774 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.746597    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerDied","Data":"91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238"}
Jul 13 20:40:26.746774 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.746704    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b" event={"ID":"2e576447-f7a9-4f98-bcf9-fd287d21acd7","Type":"ContainerDied","Data":"ed5195f3205d6b4ac5995e7a6ee14680230550583c00e6c368e5663cd1d31f4b"}
Jul 13 20:40:26.746774 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.746740    2551 scope.go:117] "RemoveContainer" containerID="4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988"
Jul 13 20:40:26.754768 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.754736    2551 scope.go:117] "RemoveContainer" containerID="91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238"
Jul 13 20:40:26.761845 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.761824    2551 scope.go:117] "RemoveContainer" containerID="e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb"
Jul 13 20:40:26.768203 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.768186    2551 scope.go:117] "RemoveContainer" containerID="4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988"
Jul 13 20:40:26.768454 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:26.768433    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988\": container with ID starting with 4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988 not found: ID does not exist" containerID="4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988"
Jul 13 20:40:26.768508 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.768462    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988"} err="failed to get container status \"4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988\": rpc error: code = NotFound desc = could not find container \"4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988\": container with ID starting with 4ee70c79147faf3d9f4b76aef5cea17f8229c6a544a91017f81c214f53d55988 not found: ID does not exist"
Jul 13 20:40:26.768508 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.768478    2551 scope.go:117] "RemoveContainer" containerID="91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238"
Jul 13 20:40:26.768725 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:26.768700    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238\": container with ID starting with 91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238 not found: ID does not exist" containerID="91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238"
Jul 13 20:40:26.768832 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.768730    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238"} err="failed to get container status \"91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238\": rpc error: code = NotFound desc = could not find container \"91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238\": container with ID starting with 91d8360f85b3918d1a301fb42e2a7c2dda288d9da79f4a0fbdde88ff80c03238 not found: ID does not exist"
Jul 13 20:40:26.768832 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.768752    2551 scope.go:117] "RemoveContainer" containerID="e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb"
Jul 13 20:40:26.769004 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:26.768987    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb\": container with ID starting with e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb not found: ID does not exist" containerID="e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb"
Jul 13 20:40:26.769047 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.769010    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb"} err="failed to get container status \"e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb\": rpc error: code = NotFound desc = could not find container \"e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb\": container with ID starting with e69fd4baca6e932c96df3c4ac28d5e8eab3cce21ab38d8698e8fc0c30274ccdb not found: ID does not exist"
Jul 13 20:40:26.773725 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.773694    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"]
Jul 13 20:40:26.776460 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.776442    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-fa5c0a-predictor-5cd7b7f947-7jn9b"]
Jul 13 20:40:26.793157 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.793138    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:26.800965 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.800940    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"]
Jul 13 20:40:26.801219 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801205    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fc1166-d332-419f-843e-043029fc065f" containerName="storage-initializer"
Jul 13 20:40:26.801304 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801223    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fc1166-d332-419f-843e-043029fc065f" containerName="storage-initializer"
Jul 13 20:40:26.801304 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801236    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="storage-initializer"
Jul 13 20:40:26.801304 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801246    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="storage-initializer"
Jul 13 20:40:26.801304 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801256    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container"
Jul 13 20:40:26.801304 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801264    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container"
Jul 13 20:40:26.801304 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801275    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kube-rbac-proxy"
Jul 13 20:40:26.801304 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801283    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kube-rbac-proxy"
Jul 13 20:40:26.801630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801348    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kserve-container"
Jul 13 20:40:26.801630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801360    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" containerName="kube-rbac-proxy"
Jul 13 20:40:26.801630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801370    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fc1166-d332-419f-843e-043029fc065f" containerName="storage-initializer"
Jul 13 20:40:26.801630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801380    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fc1166-d332-419f-843e-043029fc065f" containerName="storage-initializer"
Jul 13 20:40:26.801630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801458    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fc1166-d332-419f-843e-043029fc065f" containerName="storage-initializer"
Jul 13 20:40:26.801630 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.801468    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fc1166-d332-419f-843e-043029fc065f" containerName="storage-initializer"
Jul 13 20:40:26.805701 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.805684    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:26.808153 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.808127    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9eb9ee-kube-rbac-proxy-sar-config\""
Jul 13 20:40:26.808240 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.808163    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-9eb9ee-dockercfg-qhv6w\""
Jul 13 20:40:26.808746 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.808730    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9eb9ee-predictor-serving-cert\""
Jul 13 20:40:26.808881 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.808857    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 20:40:26.808991 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.808930    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 20:40:26.809057 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.809002    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-9eb9ee\""
Jul 13 20:40:26.809057 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.809022    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 20:40:26.819380 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.819357    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"]
Jul 13 20:40:26.895151 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.895123    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:26.910446 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.910423    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/565f7782-cf05-4c49-a0fc-e0395250dc30-proxy-tls\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:26.910566 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.910456    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:26.910566 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.910478    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565f7782-cf05-4c49-a0fc-e0395250dc30-kserve-provision-location\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:26.910566 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.910495    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-cabundle-cert\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:26.910737 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:26.910611    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl67n\" (UniqueName: \"kubernetes.io/projected/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-api-access-pl67n\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.011259 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.011221    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.011259 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.011260    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565f7782-cf05-4c49-a0fc-e0395250dc30-kserve-provision-location\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.011522 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.011283    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-cabundle-cert\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.011522 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.011311    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl67n\" (UniqueName: \"kubernetes.io/projected/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-api-access-pl67n\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.011522 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.011349    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/565f7782-cf05-4c49-a0fc-e0395250dc30-proxy-tls\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.011784 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.011731    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565f7782-cf05-4c49-a0fc-e0395250dc30-kserve-provision-location\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.012116 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.012094    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.012154 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.012094    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-cabundle-cert\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.013898 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.013871    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/565f7782-cf05-4c49-a0fc-e0395250dc30-proxy-tls\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.020733 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.020712    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl67n\" (UniqueName: \"kubernetes.io/projected/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-api-access-pl67n\") pod \"isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") " pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.115353 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.115311    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:27.136118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.136083    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e576447-f7a9-4f98-bcf9-fd287d21acd7" path="/var/lib/kubelet/pods/2e576447-f7a9-4f98-bcf9-fd287d21acd7/volumes"
Jul 13 20:40:27.259397 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.259357    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"]
Jul 13 20:40:27.263920 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:40:27.263895    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565f7782_cf05_4c49_a0fc_e0395250dc30.slice/crio-cbce8b6dc4d433a39ba26a966d489fb655e81686247d48b94743fbc7c62c615e WatchSource:0}: Error finding container cbce8b6dc4d433a39ba26a966d489fb655e81686247d48b94743fbc7c62c615e: Status 404 returned error can't find the container with id cbce8b6dc4d433a39ba26a966d489fb655e81686247d48b94743fbc7c62c615e
Jul 13 20:40:27.752614 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.752571    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2" event={"ID":"565f7782-cf05-4c49-a0fc-e0395250dc30","Type":"ContainerStarted","Data":"020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba"}
Jul 13 20:40:27.753107 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:27.752637    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2" event={"ID":"565f7782-cf05-4c49-a0fc-e0395250dc30","Type":"ContainerStarted","Data":"cbce8b6dc4d433a39ba26a966d489fb655e81686247d48b94743fbc7c62c615e"}
Jul 13 20:40:31.768609 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:31.768583    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_565f7782-cf05-4c49-a0fc-e0395250dc30/storage-initializer/0.log"
Jul 13 20:40:31.768995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:31.768621    2551 generic.go:356] "Generic (PLEG): container finished" podID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerID="020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba" exitCode=1
Jul 13 20:40:31.768995 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:31.768706    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2" event={"ID":"565f7782-cf05-4c49-a0fc-e0395250dc30","Type":"ContainerDied","Data":"020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba"}
Jul 13 20:40:32.774895 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:32.774867    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_565f7782-cf05-4c49-a0fc-e0395250dc30/storage-initializer/0.log"
Jul 13 20:40:32.775375 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:32.774921    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2" event={"ID":"565f7782-cf05-4c49-a0fc-e0395250dc30","Type":"ContainerStarted","Data":"62cb4e120b2da9708986e827fba726e694043374b10e31b969a87ff59baaa5a2"}
Jul 13 20:40:34.784627 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:34.784600    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_565f7782-cf05-4c49-a0fc-e0395250dc30/storage-initializer/1.log"
Jul 13 20:40:34.785022 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:34.785011    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_565f7782-cf05-4c49-a0fc-e0395250dc30/storage-initializer/0.log"
Jul 13 20:40:34.785083 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:34.785040    2551 generic.go:356] "Generic (PLEG): container finished" podID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerID="62cb4e120b2da9708986e827fba726e694043374b10e31b969a87ff59baaa5a2" exitCode=1
Jul 13 20:40:34.785083 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:34.785065    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2" event={"ID":"565f7782-cf05-4c49-a0fc-e0395250dc30","Type":"ContainerDied","Data":"62cb4e120b2da9708986e827fba726e694043374b10e31b969a87ff59baaa5a2"}
Jul 13 20:40:34.785143 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:34.785114    2551 scope.go:117] "RemoveContainer" containerID="020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba"
Jul 13 20:40:34.785537 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:34.785506    2551 scope.go:117] "RemoveContainer" containerID="020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba"
Jul 13 20:40:34.795028 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:34.795004    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_kserve-ci-e2e-test(565f7782-cf05-4c49-a0fc-e0395250dc30)\"" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30"
Jul 13 20:40:34.795125 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:34.795102    2551 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_kserve-ci-e2e-test_565f7782-cf05-4c49-a0fc-e0395250dc30_0 in pod sandbox cbce8b6dc4d433a39ba26a966d489fb655e81686247d48b94743fbc7c62c615e from index: no such id: '020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba'" containerID="020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba"
Jul 13 20:40:34.795191 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:34.795133    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_kserve-ci-e2e-test_565f7782-cf05-4c49-a0fc-e0395250dc30_0 in pod sandbox cbce8b6dc4d433a39ba26a966d489fb655e81686247d48b94743fbc7c62c615e from index: no such id: '020d1de0ff23cea53a30f5c31c62c187dcdacaf805acf1ba91478635a35697ba'"
Jul 13 20:40:35.789831 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:35.789801    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_565f7782-cf05-4c49-a0fc-e0395250dc30/storage-initializer/1.log"
Jul 13 20:40:36.828847 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:36.828814    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"]
Jul 13 20:40:36.893979 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:36.893937    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:36.949439 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:36.949417    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_565f7782-cf05-4c49-a0fc-e0395250dc30/storage-initializer/1.log"
Jul 13 20:40:36.952873 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:36.952853    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:37.085500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085406    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-cabundle-cert\") pod \"565f7782-cf05-4c49-a0fc-e0395250dc30\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") "
Jul 13 20:40:37.085500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085448    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl67n\" (UniqueName: \"kubernetes.io/projected/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-api-access-pl67n\") pod \"565f7782-cf05-4c49-a0fc-e0395250dc30\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") "
Jul 13 20:40:37.085500 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085502    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565f7782-cf05-4c49-a0fc-e0395250dc30-kserve-provision-location\") pod \"565f7782-cf05-4c49-a0fc-e0395250dc30\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") "
Jul 13 20:40:37.085812 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085520    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/565f7782-cf05-4c49-a0fc-e0395250dc30-proxy-tls\") pod \"565f7782-cf05-4c49-a0fc-e0395250dc30\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") "
Jul 13 20:40:37.085812 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085540    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-rbac-proxy-sar-config\") pod \"565f7782-cf05-4c49-a0fc-e0395250dc30\" (UID: \"565f7782-cf05-4c49-a0fc-e0395250dc30\") "
Jul 13 20:40:37.085812 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085772    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565f7782-cf05-4c49-a0fc-e0395250dc30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "565f7782-cf05-4c49-a0fc-e0395250dc30" (UID: "565f7782-cf05-4c49-a0fc-e0395250dc30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:40:37.085931 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085866    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "565f7782-cf05-4c49-a0fc-e0395250dc30" (UID: "565f7782-cf05-4c49-a0fc-e0395250dc30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:37.085994 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.085974    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "565f7782-cf05-4c49-a0fc-e0395250dc30" (UID: "565f7782-cf05-4c49-a0fc-e0395250dc30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:40:37.087565 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.087538    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565f7782-cf05-4c49-a0fc-e0395250dc30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "565f7782-cf05-4c49-a0fc-e0395250dc30" (UID: "565f7782-cf05-4c49-a0fc-e0395250dc30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:40:37.087702 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.087618    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-api-access-pl67n" (OuterVolumeSpecName: "kube-api-access-pl67n") pod "565f7782-cf05-4c49-a0fc-e0395250dc30" (UID: "565f7782-cf05-4c49-a0fc-e0395250dc30"). InnerVolumeSpecName "kube-api-access-pl67n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:40:37.186633 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.186605    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:37.186633 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.186632    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl67n\" (UniqueName: \"kubernetes.io/projected/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-api-access-pl67n\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:37.186633 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.186642    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/565f7782-cf05-4c49-a0fc-e0395250dc30-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:37.186862 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.186672    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/565f7782-cf05-4c49-a0fc-e0395250dc30-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:37.186862 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.186683    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/565f7782-cf05-4c49-a0fc-e0395250dc30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:40:37.798020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.797995    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2_565f7782-cf05-4c49-a0fc-e0395250dc30/storage-initializer/1.log"
Jul 13 20:40:37.798192 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.798040    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2" event={"ID":"565f7782-cf05-4c49-a0fc-e0395250dc30","Type":"ContainerDied","Data":"cbce8b6dc4d433a39ba26a966d489fb655e81686247d48b94743fbc7c62c615e"}
Jul 13 20:40:37.798192 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.798086    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"
Jul 13 20:40:37.798192 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.798097    2551 scope.go:117] "RemoveContainer" containerID="62cb4e120b2da9708986e827fba726e694043374b10e31b969a87ff59baaa5a2"
Jul 13 20:40:37.836560 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.836531    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"]
Jul 13 20:40:37.841528 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:37.841507    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9eb9ee-predictor-76d6994bb4-pjns2"]
Jul 13 20:40:39.135932 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:39.135896    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30" path="/var/lib/kubelet/pods/565f7782-cf05-4c49-a0fc-e0395250dc30/volumes"
Jul 13 20:40:42.093093 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.093047    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:42.175530 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.175489    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"]
Jul 13 20:40:42.175828 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.175812    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerName="storage-initializer"
Jul 13 20:40:42.175914 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.175832    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerName="storage-initializer"
Jul 13 20:40:42.175914 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.175860    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerName="storage-initializer"
Jul 13 20:40:42.175914 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.175869    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerName="storage-initializer"
Jul 13 20:40:42.176076 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.175937    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerName="storage-initializer"
Jul 13 20:40:42.176076 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.175950    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="565f7782-cf05-4c49-a0fc-e0395250dc30" containerName="storage-initializer"
Jul 13 20:40:42.180130 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.180107    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.190018 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.190000    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 20:40:42.190256 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.190244    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 20:40:42.190307 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.190270    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-0bd7c-predictor-serving-cert\""
Jul 13 20:40:42.190409 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.190389    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-0bd7c-kube-rbac-proxy-sar-config\""
Jul 13 20:40:42.191125 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.191109    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:40:42.191180 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.191135    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d8krp\""
Jul 13 20:40:42.195705 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.195685    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 20:40:42.197612 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.197594    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"]
Jul 13 20:40:42.320154 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.320116    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.320332 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.320160    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9887829-76cf-4302-9859-33eb369d8bb7-kserve-provision-location\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.320332 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.320184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-cabundle-cert\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.320332 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.320255    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.320332 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.320285    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrkr\" (UniqueName: \"kubernetes.io/projected/d9887829-76cf-4302-9859-33eb369d8bb7-kube-api-access-xvrkr\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.420638 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.420550    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9887829-76cf-4302-9859-33eb369d8bb7-kserve-provision-location\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.420638 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.420590    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-cabundle-cert\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.420638 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.420621    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.420932 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.420646    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrkr\" (UniqueName: \"kubernetes.io/projected/d9887829-76cf-4302-9859-33eb369d8bb7-kube-api-access-xvrkr\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.420932 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.420729    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.420932 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:42.420800    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-serving-cert: secret "raw-sklearn-0bd7c-predictor-serving-cert" not found
Jul 13 20:40:42.420932 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:40:42.420874    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls podName:d9887829-76cf-4302-9859-33eb369d8bb7 nodeName:}" failed. No retries permitted until 2026-07-13 20:40:42.920853218 +0000 UTC m=+1300.393081976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls") pod "raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" (UID: "d9887829-76cf-4302-9859-33eb369d8bb7") : secret "raw-sklearn-0bd7c-predictor-serving-cert" not found
Jul 13 20:40:42.421169 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.420965    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9887829-76cf-4302-9859-33eb369d8bb7-kserve-provision-location\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.421450 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.421420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.421702 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.421687    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-cabundle-cert\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.432235 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.432214    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrkr\" (UniqueName: \"kubernetes.io/projected/d9887829-76cf-4302-9859-33eb369d8bb7-kube-api-access-xvrkr\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.691979 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.691892    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:40:42.923564 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.923525    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:42.925902 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:42.925882    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls\") pod \"raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") " pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:43.089753 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:43.089667    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:43.217678 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:43.217627    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"]
Jul 13 20:40:43.220954 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:40:43.220928    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9887829_76cf_4302_9859_33eb369d8bb7.slice/crio-48f09fb5ada13e488510527f1c7c1e0116cd7a278c675fd37bab81363e3be21e WatchSource:0}: Error finding container 48f09fb5ada13e488510527f1c7c1e0116cd7a278c675fd37bab81363e3be21e: Status 404 returned error can't find the container with id 48f09fb5ada13e488510527f1c7c1e0116cd7a278c675fd37bab81363e3be21e
Jul 13 20:40:43.820935 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:43.820903    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerStarted","Data":"22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5"}
Jul 13 20:40:43.821146 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:43.820957    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerStarted","Data":"48f09fb5ada13e488510527f1c7c1e0116cd7a278c675fd37bab81363e3be21e"}
Jul 13 20:40:47.836403 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:47.836364    2551 generic.go:356] "Generic (PLEG): container finished" podID="d9887829-76cf-4302-9859-33eb369d8bb7" containerID="22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5" exitCode=0
Jul 13 20:40:47.836808 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:47.836439    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerDied","Data":"22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5"}
Jul 13 20:40:48.842061 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:48.842030    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerStarted","Data":"6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711"}
Jul 13 20:40:48.842443 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:48.842094    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerStarted","Data":"d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5"}
Jul 13 20:40:48.842443 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:48.842277    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:48.871903 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:48.871856    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podStartSLOduration=3.058171982 podStartE2EDuration="6.871841156s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813669174s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:40:48.871489305 +0000 UTC m=+1306.343718084" watchObservedRunningTime="2026-07-13 20:40:48.871841156 +0000 UTC m=+1306.344069930"
Jul 13 20:40:49.846186 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:49.846151    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:49.847285 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:49.847260    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:40:50.849953 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:50.849916    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:40:55.854532 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:55.854501    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:40:55.855190 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:40:55.855161    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:41:05.856026 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:41:05.855985    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:41:15.856019 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:41:15.855976    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:41:25.855692 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:41:25.855591    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:41:35.855720 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:41:35.855674    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:41:45.855345 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:41:45.855305    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:41:55.855695 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:41:55.855635    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:42:05.855791 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:05.855760    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:42:11.973799 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:11.973772    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:12.254309 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:12.254226    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"]
Jul 13 20:42:12.254477 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:12.254451    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" containerID="cri-o://d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5" gracePeriod=30
Jul 13 20:42:12.254549 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:12.254506    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kube-rbac-proxy" containerID="cri-o://6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711" gracePeriod=30
Jul 13 20:42:12.267118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:12.267094    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:13.140577 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:13.140541    2551 generic.go:356] "Generic (PLEG): container finished" podID="d9887829-76cf-4302-9859-33eb369d8bb7" containerID="6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711" exitCode=2
Jul 13 20:42:13.140968 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:13.140610    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerDied","Data":"6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711"}
Jul 13 20:42:15.850449 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:15.850408    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 13 20:42:15.855785 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:15.855755    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 13 20:42:16.893155 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.893131    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:42:16.994210 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994118    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9887829-76cf-4302-9859-33eb369d8bb7-kserve-provision-location\") pod \"d9887829-76cf-4302-9859-33eb369d8bb7\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") "
Jul 13 20:42:16.994210 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994195    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-cabundle-cert\") pod \"d9887829-76cf-4302-9859-33eb369d8bb7\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") "
Jul 13 20:42:16.994434 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994236    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-kube-rbac-proxy-sar-config\") pod \"d9887829-76cf-4302-9859-33eb369d8bb7\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") "
Jul 13 20:42:16.994434 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994289    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls\") pod \"d9887829-76cf-4302-9859-33eb369d8bb7\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") "
Jul 13 20:42:16.994434 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994314    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvrkr\" (UniqueName: \"kubernetes.io/projected/d9887829-76cf-4302-9859-33eb369d8bb7-kube-api-access-xvrkr\") pod \"d9887829-76cf-4302-9859-33eb369d8bb7\" (UID: \"d9887829-76cf-4302-9859-33eb369d8bb7\") "
Jul 13 20:42:16.994587 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994521    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9887829-76cf-4302-9859-33eb369d8bb7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9887829-76cf-4302-9859-33eb369d8bb7" (UID: "d9887829-76cf-4302-9859-33eb369d8bb7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:42:16.994680 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994627    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9887829-76cf-4302-9859-33eb369d8bb7" (UID: "d9887829-76cf-4302-9859-33eb369d8bb7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:42:16.994680 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.994647    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9887829-76cf-4302-9859-33eb369d8bb7" (UID: "d9887829-76cf-4302-9859-33eb369d8bb7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:42:16.996428 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.996399    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9887829-76cf-4302-9859-33eb369d8bb7-kube-api-access-xvrkr" (OuterVolumeSpecName: "kube-api-access-xvrkr") pod "d9887829-76cf-4302-9859-33eb369d8bb7" (UID: "d9887829-76cf-4302-9859-33eb369d8bb7"). InnerVolumeSpecName "kube-api-access-xvrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:42:16.996546 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:16.996511    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9887829-76cf-4302-9859-33eb369d8bb7" (UID: "d9887829-76cf-4302-9859-33eb369d8bb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:42:17.094919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.094884    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:42:17.094919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.094913    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9887829-76cf-4302-9859-33eb369d8bb7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:42:17.094919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.094924    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9887829-76cf-4302-9859-33eb369d8bb7-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:42:17.095134 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.094935    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvrkr\" (UniqueName: \"kubernetes.io/projected/d9887829-76cf-4302-9859-33eb369d8bb7-kube-api-access-xvrkr\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:42:17.095134 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.094943    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9887829-76cf-4302-9859-33eb369d8bb7-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:42:17.156785 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.156753    2551 generic.go:356] "Generic (PLEG): container finished" podID="d9887829-76cf-4302-9859-33eb369d8bb7" containerID="d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5" exitCode=0
Jul 13 20:42:17.156919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.156807    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerDied","Data":"d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5"}
Jul 13 20:42:17.156919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.156834    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"
Jul 13 20:42:17.156919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.156867    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq" event={"ID":"d9887829-76cf-4302-9859-33eb369d8bb7","Type":"ContainerDied","Data":"48f09fb5ada13e488510527f1c7c1e0116cd7a278c675fd37bab81363e3be21e"}
Jul 13 20:42:17.156919 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.156903    2551 scope.go:117] "RemoveContainer" containerID="6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711"
Jul 13 20:42:17.164699 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.164683    2551 scope.go:117] "RemoveContainer" containerID="d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5"
Jul 13 20:42:17.171579 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.171562    2551 scope.go:117] "RemoveContainer" containerID="22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5"
Jul 13 20:42:17.178430 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.178404    2551 scope.go:117] "RemoveContainer" containerID="6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711"
Jul 13 20:42:17.178845 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:42:17.178822    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711\": container with ID starting with 6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711 not found: ID does not exist" containerID="6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711"
Jul 13 20:42:17.179080 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.178859    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711"} err="failed to get container status \"6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711\": rpc error: code = NotFound desc = could not find container \"6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711\": container with ID starting with 6a2465085f4477d68ae8799760ba0ded26555082ceae4837b6223e04a01ae711 not found: ID does not exist"
Jul 13 20:42:17.179080 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.178881    2551 scope.go:117] "RemoveContainer" containerID="d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5"
Jul 13 20:42:17.179215 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:42:17.179198    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5\": container with ID starting with d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5 not found: ID does not exist" containerID="d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5"
Jul 13 20:42:17.179282 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.179221    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5"} err="failed to get container status \"d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5\": rpc error: code = NotFound desc = could not find container \"d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5\": container with ID starting with d38da2bd9ea78938d27f991854a90cea40a00d99c040b63bf190086fa17297f5 not found: ID does not exist"
Jul 13 20:42:17.179282 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.179238    2551 scope.go:117] "RemoveContainer" containerID="22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5"
Jul 13 20:42:17.179465 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:42:17.179447    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5\": container with ID starting with 22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5 not found: ID does not exist" containerID="22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5"
Jul 13 20:42:17.179527 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.179478    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5"} err="failed to get container status \"22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5\": rpc error: code = NotFound desc = could not find container \"22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5\": container with ID starting with 22104f8c1abc47d822352099b528b377e5af368dd8f5566d5a4383be2c75d5a5 not found: ID does not exist"
Jul 13 20:42:17.179589 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.179531    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"]
Jul 13 20:42:17.186015 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.185992    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-0bd7c-predictor-648f8cc8f8-bb7tq"]
Jul 13 20:42:17.390882 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.390853    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:17.513916 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.513877    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"]
Jul 13 20:42:17.514238 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514223    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container"
Jul 13 20:42:17.514283 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514240    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container"
Jul 13 20:42:17.514283 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514261    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kube-rbac-proxy"
Jul 13 20:42:17.514283 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514269    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kube-rbac-proxy"
Jul 13 20:42:17.514379 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514283    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="storage-initializer"
Jul 13 20:42:17.514379 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514292    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="storage-initializer"
Jul 13 20:42:17.514379 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514349    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kube-rbac-proxy"
Jul 13 20:42:17.514379 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.514361    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" containerName="kserve-container"
Jul 13 20:42:17.518611 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.518592    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.523673 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.523634    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-d316d-predictor-serving-cert\""
Jul 13 20:42:17.523916 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.523895    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 20:42:17.524226 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.524207    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-d8krp\""
Jul 13 20:42:17.524298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.524224    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-d316d-kube-rbac-proxy-sar-config\""
Jul 13 20:42:17.524298 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.524248    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:42:17.524436 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.524420    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 20:42:17.524599 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.524584    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 20:42:17.539321 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.539300    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"]
Jul 13 20:42:17.599391 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.599359    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzj6\" (UniqueName: \"kubernetes.io/projected/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-api-access-nwzj6\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.599391 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.599394    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c74f8a00-4f74-4e52-bc85-3014deea90fd-kserve-provision-location\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.599623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.599430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.599623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.599523    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-cabundle-cert\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.599623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.599566    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.700876 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.700778    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.700876 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.700839    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-cabundle-cert\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.701118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.700874    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.701118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.700928    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzj6\" (UniqueName: \"kubernetes.io/projected/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-api-access-nwzj6\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.701118 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.700953    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c74f8a00-4f74-4e52-bc85-3014deea90fd-kserve-provision-location\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.701118 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:42:17.700962    2551 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-serving-cert: secret "raw-sklearn-runtime-d316d-predictor-serving-cert" not found
Jul 13 20:42:17.701118 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:42:17.701096    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls podName:c74f8a00-4f74-4e52-bc85-3014deea90fd nodeName:}" failed. No retries permitted until 2026-07-13 20:42:18.201017231 +0000 UTC m=+1395.673245991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls") pod "raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" (UID: "c74f8a00-4f74-4e52-bc85-3014deea90fd") : secret "raw-sklearn-runtime-d316d-predictor-serving-cert" not found
Jul 13 20:42:17.701385 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.701340    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c74f8a00-4f74-4e52-bc85-3014deea90fd-kserve-provision-location\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.701633 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.701609    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.701694 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.701609    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-cabundle-cert\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.713969 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.713941    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzj6\" (UniqueName: \"kubernetes.io/projected/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-api-access-nwzj6\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:17.799566 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:17.799519    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:42:18.203926 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:18.203885    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:18.206306 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:18.206279    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls\") pod \"raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:18.428783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:18.428742    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:18.558409 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:18.558247    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"]
Jul 13 20:42:18.561013 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:42:18.560985    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74f8a00_4f74_4e52_bc85_3014deea90fd.slice/crio-f3149bb33b277553ff02fa827b1b607cbd206b1c6ce9d1df6912aeaa94ca6959 WatchSource:0}: Error finding container f3149bb33b277553ff02fa827b1b607cbd206b1c6ce9d1df6912aeaa94ca6959: Status 404 returned error can't find the container with id f3149bb33b277553ff02fa827b1b607cbd206b1c6ce9d1df6912aeaa94ca6959
Jul 13 20:42:19.136313 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:19.136283    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9887829-76cf-4302-9859-33eb369d8bb7" path="/var/lib/kubelet/pods/d9887829-76cf-4302-9859-33eb369d8bb7/volumes"
Jul 13 20:42:19.165543 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:19.165508    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerStarted","Data":"49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e"}
Jul 13 20:42:19.165731 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:19.165564    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerStarted","Data":"f3149bb33b277553ff02fa827b1b607cbd206b1c6ce9d1df6912aeaa94ca6959"}
Jul 13 20:42:23.180672 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:23.180627    2551 generic.go:356] "Generic (PLEG): container finished" podID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerID="49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e" exitCode=0
Jul 13 20:42:23.181047 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:23.180694    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerDied","Data":"49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e"}
Jul 13 20:42:24.185901 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:24.185865    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerStarted","Data":"c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894"}
Jul 13 20:42:24.186288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:24.185956    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerStarted","Data":"ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699"}
Jul 13 20:42:24.186288 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:24.186234    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:24.205045 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:24.204997    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podStartSLOduration=3.514479928 podStartE2EDuration="7.204979966s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.690500038s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:42:24.204331631 +0000 UTC m=+1401.676560405" watchObservedRunningTime="2026-07-13 20:42:24.204979966 +0000 UTC m=+1401.677208740"
Jul 13 20:42:25.189352 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:25.189321    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:25.190594 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:25.190570    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:42:26.193020 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:26.192977    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:42:31.197902 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:31.197875    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:42:31.198380 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:31.198353    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:42:41.198476 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:41.198438    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:42:51.198956 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:42:51.198916    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:43:01.199086 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:01.199001    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:43:11.198905 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:11.198862    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:43:21.198457 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:21.198422    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:43:31.198944 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:31.198908    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:43:41.198816 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:41.198779    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:43:47.565593 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:47.565559    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"]
Jul 13 20:43:47.566042 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:47.565824    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" containerID="cri-o://ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699" gracePeriod=30
Jul 13 20:43:47.566042 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:47.565887    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kube-rbac-proxy" containerID="cri-o://c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894" gracePeriod=30
Jul 13 20:43:48.398365 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:48.398328    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:48.485737 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:48.485702    2551 generic.go:356] "Generic (PLEG): container finished" podID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerID="c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894" exitCode=2
Jul 13 20:43:48.485887 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:48.485755    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerDied","Data":"c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894"}
Jul 13 20:43:51.193259 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:51.193218    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 13 20:43:51.198287 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:51.198259    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 13 20:43:52.306285 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.306258    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:43:52.395769 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.395677    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwzj6\" (UniqueName: \"kubernetes.io/projected/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-api-access-nwzj6\") pod \"c74f8a00-4f74-4e52-bc85-3014deea90fd\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") "
Jul 13 20:43:52.395769 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.395747    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls\") pod \"c74f8a00-4f74-4e52-bc85-3014deea90fd\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") "
Jul 13 20:43:52.395963 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.395782    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-rbac-proxy-sar-config\") pod \"c74f8a00-4f74-4e52-bc85-3014deea90fd\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") "
Jul 13 20:43:52.395963 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.395825    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c74f8a00-4f74-4e52-bc85-3014deea90fd-kserve-provision-location\") pod \"c74f8a00-4f74-4e52-bc85-3014deea90fd\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") "
Jul 13 20:43:52.395963 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.395852    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-cabundle-cert\") pod \"c74f8a00-4f74-4e52-bc85-3014deea90fd\" (UID: \"c74f8a00-4f74-4e52-bc85-3014deea90fd\") "
Jul 13 20:43:52.396541 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.396460    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74f8a00-4f74-4e52-bc85-3014deea90fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c74f8a00-4f74-4e52-bc85-3014deea90fd" (UID: "c74f8a00-4f74-4e52-bc85-3014deea90fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:43:52.396541 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.396489    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c74f8a00-4f74-4e52-bc85-3014deea90fd" (UID: "c74f8a00-4f74-4e52-bc85-3014deea90fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:43:52.396748 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.396544    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c74f8a00-4f74-4e52-bc85-3014deea90fd" (UID: "c74f8a00-4f74-4e52-bc85-3014deea90fd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 20:43:52.399200 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.399144    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c74f8a00-4f74-4e52-bc85-3014deea90fd" (UID: "c74f8a00-4f74-4e52-bc85-3014deea90fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:43:52.399200 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.399176    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-api-access-nwzj6" (OuterVolumeSpecName: "kube-api-access-nwzj6") pod "c74f8a00-4f74-4e52-bc85-3014deea90fd" (UID: "c74f8a00-4f74-4e52-bc85-3014deea90fd"). InnerVolumeSpecName "kube-api-access-nwzj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:43:52.497051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.497015    2551 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:43:52.497051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.497041    2551 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c74f8a00-4f74-4e52-bc85-3014deea90fd-kserve-provision-location\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:43:52.497051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.497052    2551 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c74f8a00-4f74-4e52-bc85-3014deea90fd-cabundle-cert\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:43:52.497051 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.497062    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzj6\" (UniqueName: \"kubernetes.io/projected/c74f8a00-4f74-4e52-bc85-3014deea90fd-kube-api-access-nwzj6\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:43:52.497344 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.497072    2551 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c74f8a00-4f74-4e52-bc85-3014deea90fd-proxy-tls\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:43:52.501049 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.501017    2551 generic.go:356] "Generic (PLEG): container finished" podID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerID="ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699" exitCode=0
Jul 13 20:43:52.501146 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.501073    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerDied","Data":"ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699"}
Jul 13 20:43:52.501146 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.501098    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"
Jul 13 20:43:52.501232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.501163    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2" event={"ID":"c74f8a00-4f74-4e52-bc85-3014deea90fd","Type":"ContainerDied","Data":"f3149bb33b277553ff02fa827b1b607cbd206b1c6ce9d1df6912aeaa94ca6959"}
Jul 13 20:43:52.501232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.501203    2551 scope.go:117] "RemoveContainer" containerID="c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894"
Jul 13 20:43:52.509165 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.509142    2551 scope.go:117] "RemoveContainer" containerID="ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699"
Jul 13 20:43:52.516228 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.516205    2551 scope.go:117] "RemoveContainer" containerID="49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e"
Jul 13 20:43:52.521079 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.521056    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"]
Jul 13 20:43:52.524032 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.524014    2551 scope.go:117] "RemoveContainer" containerID="c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894"
Jul 13 20:43:52.524308 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.524290    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d316d-predictor-79c76cd57d-9t7s2"]
Jul 13 20:43:52.524360 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:43:52.524290    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894\": container with ID starting with c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894 not found: ID does not exist" containerID="c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894"
Jul 13 20:43:52.524360 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.524347    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894"} err="failed to get container status \"c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894\": rpc error: code = NotFound desc = could not find container \"c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894\": container with ID starting with c0d379a9a2a5f2b899e222dc29cb7aaa961bf3ce5ac7bc01d49d176993cb7894 not found: ID does not exist"
Jul 13 20:43:52.524433 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.524367    2551 scope.go:117] "RemoveContainer" containerID="ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699"
Jul 13 20:43:52.524597 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:43:52.524580    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699\": container with ID starting with ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699 not found: ID does not exist" containerID="ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699"
Jul 13 20:43:52.524637 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.524602    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699"} err="failed to get container status \"ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699\": rpc error: code = NotFound desc = could not find container \"ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699\": container with ID starting with ecad382abd23225e271e2bb0d73c48c5914076db5fcd82b7f1be1a2f87ce6699 not found: ID does not exist"
Jul 13 20:43:52.524637 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.524617    2551 scope.go:117] "RemoveContainer" containerID="49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e"
Jul 13 20:43:52.524873 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:43:52.524855    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e\": container with ID starting with 49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e not found: ID does not exist" containerID="49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e"
Jul 13 20:43:52.524914 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:52.524880    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e"} err="failed to get container status \"49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e\": rpc error: code = NotFound desc = could not find container \"49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e\": container with ID starting with 49031f282211d99a791f582f4382f27af1a43526d05d69fe9db92d2337c2027e not found: ID does not exist"
Jul 13 20:43:53.135922 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.135888    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" path="/var/lib/kubelet/pods/c74f8a00-4f74-4e52-bc85-3014deea90fd/volumes"
Jul 13 20:43:53.573230 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573144    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-257d9/must-gather-jj6cx"]
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573414    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kube-rbac-proxy"
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573426    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kube-rbac-proxy"
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573436    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="storage-initializer"
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573441    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="storage-initializer"
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573450    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container"
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573456    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container"
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573507    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kube-rbac-proxy"
Jul 13 20:43:53.573623 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.573515    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74f8a00-4f74-4e52-bc85-3014deea90fd" containerName="kserve-container"
Jul 13 20:43:53.577549 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.577533    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:53.579575 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.579554    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-257d9\"/\"kube-root-ca.crt\""
Jul 13 20:43:53.580076 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.580060    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-257d9\"/\"default-dockercfg-trlt2\""
Jul 13 20:43:53.580157 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.580059    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-257d9\"/\"openshift-service-ca.crt\""
Jul 13 20:43:53.584098 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.584077    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-257d9/must-gather-jj6cx"]
Jul 13 20:43:53.584185 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.584076    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:53.604079 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.604051    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9s7\" (UniqueName: \"kubernetes.io/projected/754805df-70b9-4010-9236-1c7f2472bc47-kube-api-access-ng9s7\") pod \"must-gather-jj6cx\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") " pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:53.604207 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.604108    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754805df-70b9-4010-9236-1c7f2472bc47-must-gather-output\") pod \"must-gather-jj6cx\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") " pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:53.677528 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.677496    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:43:53.705272 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.705248    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9s7\" (UniqueName: \"kubernetes.io/projected/754805df-70b9-4010-9236-1c7f2472bc47-kube-api-access-ng9s7\") pod \"must-gather-jj6cx\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") " pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:53.705397 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.705294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754805df-70b9-4010-9236-1c7f2472bc47-must-gather-output\") pod \"must-gather-jj6cx\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") " pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:53.705562 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.705548    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754805df-70b9-4010-9236-1c7f2472bc47-must-gather-output\") pod \"must-gather-jj6cx\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") " pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:53.712285 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.712259    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9s7\" (UniqueName: \"kubernetes.io/projected/754805df-70b9-4010-9236-1c7f2472bc47-kube-api-access-ng9s7\") pod \"must-gather-jj6cx\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") " pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:53.886228 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:53.886202    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:43:54.010505 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:54.010480    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-257d9/must-gather-jj6cx"]
Jul 13 20:43:54.012734 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:43:54.012708    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod754805df_70b9_4010_9236_1c7f2472bc47.slice/crio-f12bca1aaefb795fdfb54bedc454b8e12310c268f2a43b5d2e052e309abef99b WatchSource:0}: Error finding container f12bca1aaefb795fdfb54bedc454b8e12310c268f2a43b5d2e052e309abef99b: Status 404 returned error can't find the container with id f12bca1aaefb795fdfb54bedc454b8e12310c268f2a43b5d2e052e309abef99b
Jul 13 20:43:54.014569 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:54.014552    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 20:43:54.509168 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:43:54.509135    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-257d9/must-gather-jj6cx" event={"ID":"754805df-70b9-4010-9236-1c7f2472bc47","Type":"ContainerStarted","Data":"f12bca1aaefb795fdfb54bedc454b8e12310c268f2a43b5d2e052e309abef99b"}
Jul 13 20:44:01.545978 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:01.545937    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-257d9/must-gather-jj6cx" event={"ID":"754805df-70b9-4010-9236-1c7f2472bc47","Type":"ContainerStarted","Data":"2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542"}
Jul 13 20:44:01.546384 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:01.545986    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-257d9/must-gather-jj6cx" event={"ID":"754805df-70b9-4010-9236-1c7f2472bc47","Type":"ContainerStarted","Data":"aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3"}
Jul 13 20:44:01.560960 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:01.560910    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-257d9/must-gather-jj6cx" podStartSLOduration=1.757113894 podStartE2EDuration="8.560893992s" totalImagesPullingTime="6.803780098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:44:01.560146559 +0000 UTC m=+1499.032375334" watchObservedRunningTime="2026-07-13 20:44:01.560893992 +0000 UTC m=+1499.033122765"
Jul 13 20:44:03.020139 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:03.020110    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:44:03.021398 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:03.021377    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:44:22.632819 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:22.632788    2551 generic.go:356] "Generic (PLEG): container finished" podID="754805df-70b9-4010-9236-1c7f2472bc47" containerID="aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3" exitCode=0
Jul 13 20:44:22.633226 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:22.632864    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-257d9/must-gather-jj6cx" event={"ID":"754805df-70b9-4010-9236-1c7f2472bc47","Type":"ContainerDied","Data":"aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3"}
Jul 13 20:44:22.633226 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:22.633203    2551 scope.go:117] "RemoveContainer" containerID="aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3"
Jul 13 20:44:22.833157 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:22.833127    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-257d9_must-gather-jj6cx_754805df-70b9-4010-9236-1c7f2472bc47/gather/0.log"
Jul 13 20:44:23.225783 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:23.225751    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:24.094109 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:24.094032    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:24.195679 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:24.195637    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:26.173553 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:26.173521    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vjwb9_5bf83bc3-42e1-409e-a511-bce2009ba373/global-pull-secret-syncer/0.log"
Jul 13 20:44:26.218136 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:26.218103    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log"
Jul 13 20:44:26.341109 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:26.341078    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-42.ec2.internal_e08333fc818b35b3d22fa5e3da933609/haproxy/0.log"
Jul 13 20:44:28.205332 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.205296    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-257d9/must-gather-jj6cx"]
Jul 13 20:44:28.205830 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.205470    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-257d9/must-gather-jj6cx" podUID="754805df-70b9-4010-9236-1c7f2472bc47" containerName="copy" containerID="cri-o://2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542" gracePeriod=2
Jul 13 20:44:28.207280 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:44:28.207243    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jj6cx\" is forbidden: User \"system:node:ip-10-0-131-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-257d9\": no relationship found between node 'ip-10-0-131-42.ec2.internal' and this object" podUID="754805df-70b9-4010-9236-1c7f2472bc47" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:44:28.207586 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.207565    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-257d9/must-gather-jj6cx"]
Jul 13 20:44:28.429249 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.429227    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-257d9_must-gather-jj6cx_754805df-70b9-4010-9236-1c7f2472bc47/copy/0.log"
Jul 13 20:44:28.430766 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.430748    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:44:28.432372 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:44:28.432350    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jj6cx\" is forbidden: User \"system:node:ip-10-0-131-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-257d9\": no relationship found between node 'ip-10-0-131-42.ec2.internal' and this object" podUID="754805df-70b9-4010-9236-1c7f2472bc47" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:44:28.572229 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.572137    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754805df-70b9-4010-9236-1c7f2472bc47-must-gather-output\") pod \"754805df-70b9-4010-9236-1c7f2472bc47\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") "
Jul 13 20:44:28.572229 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.572183    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng9s7\" (UniqueName: \"kubernetes.io/projected/754805df-70b9-4010-9236-1c7f2472bc47-kube-api-access-ng9s7\") pod \"754805df-70b9-4010-9236-1c7f2472bc47\" (UID: \"754805df-70b9-4010-9236-1c7f2472bc47\") "
Jul 13 20:44:28.573425 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.573397    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/754805df-70b9-4010-9236-1c7f2472bc47-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "754805df-70b9-4010-9236-1c7f2472bc47" (UID: "754805df-70b9-4010-9236-1c7f2472bc47"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:44:28.574392 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.574367    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754805df-70b9-4010-9236-1c7f2472bc47-kube-api-access-ng9s7" (OuterVolumeSpecName: "kube-api-access-ng9s7") pod "754805df-70b9-4010-9236-1c7f2472bc47" (UID: "754805df-70b9-4010-9236-1c7f2472bc47"). InnerVolumeSpecName "kube-api-access-ng9s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:44:28.655148 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.655121    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-257d9_must-gather-jj6cx_754805df-70b9-4010-9236-1c7f2472bc47/copy/0.log"
Jul 13 20:44:28.655495 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.655474    2551 generic.go:356] "Generic (PLEG): container finished" podID="754805df-70b9-4010-9236-1c7f2472bc47" containerID="2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542" exitCode=143
Jul 13 20:44:28.655569 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.655533    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:44:28.655569 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.655562    2551 scope.go:117] "RemoveContainer" containerID="2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542"
Jul 13 20:44:28.657346 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:44:28.657325    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jj6cx\" is forbidden: User \"system:node:ip-10-0-131-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-257d9\": no relationship found between node 'ip-10-0-131-42.ec2.internal' and this object" podUID="754805df-70b9-4010-9236-1c7f2472bc47" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:44:28.662798 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.662780    2551 scope.go:117] "RemoveContainer" containerID="aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3"
Jul 13 20:44:28.665235 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:44:28.665213    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-jj6cx\" is forbidden: User \"system:node:ip-10-0-131-42.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-257d9\": no relationship found between node 'ip-10-0-131-42.ec2.internal' and this object" podUID="754805df-70b9-4010-9236-1c7f2472bc47" pod="openshift-must-gather-257d9/must-gather-jj6cx"
Jul 13 20:44:28.673548 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.673521    2551 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/754805df-70b9-4010-9236-1c7f2472bc47-must-gather-output\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:44:28.673548 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.673544    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9s7\" (UniqueName: \"kubernetes.io/projected/754805df-70b9-4010-9236-1c7f2472bc47-kube-api-access-ng9s7\") on node \"ip-10-0-131-42.ec2.internal\" DevicePath \"\""
Jul 13 20:44:28.674634 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.674612    2551 scope.go:117] "RemoveContainer" containerID="2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542"
Jul 13 20:44:28.674904 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:44:28.674885    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542\": container with ID starting with 2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542 not found: ID does not exist" containerID="2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542"
Jul 13 20:44:28.674986 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.674917    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542"} err="failed to get container status \"2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542\": rpc error: code = NotFound desc = could not find container \"2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542\": container with ID starting with 2488c719dc20ab85ea50b9c6036c9c6b0ffe2b9717d76f92b6027d08605b3542 not found: ID does not exist"
Jul 13 20:44:28.674986 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.674941    2551 scope.go:117] "RemoveContainer" containerID="aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3"
Jul 13 20:44:28.675193 ip-10-0-131-42 kubenswrapper[2551]: E0713 20:44:28.675177    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3\": container with ID starting with aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3 not found: ID does not exist" containerID="aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3"
Jul 13 20:44:28.675232 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.675199    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3"} err="failed to get container status \"aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3\": rpc error: code = NotFound desc = could not find container \"aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3\": container with ID starting with aef13c93a3c026a1fd15dafe6186c856fbbb575a24a0ac52bb368977bc69b6c3 not found: ID does not exist"
Jul 13 20:44:28.850970 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.850937    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:28.917289 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:28.917257    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8xhr5_a3531f22-c8c0-440e-a943-81e2490fc57d/konnectivity-agent/0.log" limit=1024
Jul 13 20:44:29.136716 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:29.136626    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="754805df-70b9-4010-9236-1c7f2472bc47" path="/var/lib/kubelet/pods/754805df-70b9-4010-9236-1c7f2472bc47/volumes"
Jul 13 20:44:31.276937 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:31.276912    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgfwr_1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7/node-exporter/0.log"
Jul 13 20:44:31.312250 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:31.312224    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgfwr_1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7/kube-rbac-proxy/0.log"
Jul 13 20:44:31.347966 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:31.347938    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgfwr_1b3fdf6a-ab13-43f6-b2a1-73b02e72a5f7/init-textfile/0.log"
Jul 13 20:44:34.251771 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.251739    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"]
Jul 13 20:44:34.252162 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.251979    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754805df-70b9-4010-9236-1c7f2472bc47" containerName="gather"
Jul 13 20:44:34.252162 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.251989    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="754805df-70b9-4010-9236-1c7f2472bc47" containerName="gather"
Jul 13 20:44:34.252162 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.251998    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="754805df-70b9-4010-9236-1c7f2472bc47" containerName="copy"
Jul 13 20:44:34.252162 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.252003    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="754805df-70b9-4010-9236-1c7f2472bc47" containerName="copy"
Jul 13 20:44:34.252162 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.252050    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="754805df-70b9-4010-9236-1c7f2472bc47" containerName="copy"
Jul 13 20:44:34.252162 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.252059    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="754805df-70b9-4010-9236-1c7f2472bc47" containerName="gather"
Jul 13 20:44:34.253762 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.253743    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.255941 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.255919    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k5kgn\"/\"openshift-service-ca.crt\""
Jul 13 20:44:34.256526 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.256511    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k5kgn\"/\"kube-root-ca.crt\""
Jul 13 20:44:34.256616 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.256522    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k5kgn\"/\"default-dockercfg-rgvhk\""
Jul 13 20:44:34.263102 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.263083    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"]
Jul 13 20:44:34.413487 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.413437    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-podres\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.413487 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.413484    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-lib-modules\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.413726 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.413519    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-sys\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.413726 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.413545    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-proc\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.413726 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.413570    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggm56\" (UniqueName: \"kubernetes.io/projected/8734f8d3-9c06-48bc-9f94-17a5774a28f4-kube-api-access-ggm56\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.514789 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514696    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-podres\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.514789 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-lib-modules\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.514789 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514769    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-sys\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.514789 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514795    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-proc\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.515002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggm56\" (UniqueName: \"kubernetes.io/projected/8734f8d3-9c06-48bc-9f94-17a5774a28f4-kube-api-access-ggm56\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.515002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514876    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-podres\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.515002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514905    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-sys\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.515002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514920    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-proc\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.515002 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.514882    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8734f8d3-9c06-48bc-9f94-17a5774a28f4-lib-modules\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.523222 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.523198    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggm56\" (UniqueName: \"kubernetes.io/projected/8734f8d3-9c06-48bc-9f94-17a5774a28f4-kube-api-access-ggm56\") pod \"perf-node-gather-daemonset-4vnsp\" (UID: \"8734f8d3-9c06-48bc-9f94-17a5774a28f4\") " pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.563276 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.563248    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:34.686273 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:34.686240    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"]
Jul 13 20:44:34.689396 ip-10-0-131-42 kubenswrapper[2551]: W0713 20:44:34.689372    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8734f8d3_9c06_48bc_9f94_17a5774a28f4.slice/crio-43c40a4e4343ed65926f3903c0287089f92caea2dd6da0ef913eb3fef2c0e3a4 WatchSource:0}: Error finding container 43c40a4e4343ed65926f3903c0287089f92caea2dd6da0ef913eb3fef2c0e3a4: Status 404 returned error can't find the container with id 43c40a4e4343ed65926f3903c0287089f92caea2dd6da0ef913eb3fef2c0e3a4
Jul 13 20:44:35.555697 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:35.555669    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqt7x_17b92591-8c5b-4fb5-ba4b-271efe211e11/dns/0.log"
Jul 13 20:44:35.578379 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:35.578358    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lqt7x_17b92591-8c5b-4fb5-ba4b-271efe211e11/kube-rbac-proxy/0.log"
Jul 13 20:44:35.679738 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:35.679706    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp" event={"ID":"8734f8d3-9c06-48bc-9f94-17a5774a28f4","Type":"ContainerStarted","Data":"0467f4c45a5f728b8108e5850d832f020191feb9baf4026d57117f11bd71bbd4"}
Jul 13 20:44:35.679916 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:35.679756    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp" event={"ID":"8734f8d3-9c06-48bc-9f94-17a5774a28f4","Type":"ContainerStarted","Data":"43c40a4e4343ed65926f3903c0287089f92caea2dd6da0ef913eb3fef2c0e3a4"}
Jul 13 20:44:35.679916 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:35.679812    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:35.694752 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:35.694705    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp" podStartSLOduration=1.694689446 podStartE2EDuration="1.694689446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:44:35.693903729 +0000 UTC m=+1533.166132508" watchObservedRunningTime="2026-07-13 20:44:35.694689446 +0000 UTC m=+1533.166918220"
Jul 13 20:44:35.742614 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:35.742582    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-js24l_2066f3e0-e084-4802-8e22-bf7eec308a63/dns-node-resolver/0.log"
Jul 13 20:44:36.216668 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:36.216625    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5hd22_7c02c555-f585-4b0f-877d-766cecd345e7/node-ca/0.log"
Jul 13 20:44:37.409876 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:37.409844    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fwtrx_b05a9784-a678-478a-9694-00408b02e4cf/serve-healthcheck-canary/0.log"
Jul 13 20:44:37.993573 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:37.993547    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdd9x_515fd3ec-14df-46c2-b4ff-5638b256d6a9/kube-rbac-proxy/0.log"
Jul 13 20:44:38.012635 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:38.012606    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdd9x_515fd3ec-14df-46c2-b4ff-5638b256d6a9/exporter/0.log"
Jul 13 20:44:38.032084 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:38.032058    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdd9x_515fd3ec-14df-46c2-b4ff-5638b256d6a9/extractor/0.log"
Jul 13 20:44:40.087520 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:40.087493    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-vwxlw_cbb99aa6-4589-4999-bd22-cd10fc524542/manager/0.log"
Jul 13 20:44:40.109570 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:40.109546    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-6nh4z_9e5fa0d7-2205-4f11-ae23-0eec5ffdcf3c/server/0.log"
Jul 13 20:44:40.280029 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:40.280003    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pkmb9_c09814fd-b698-4cbe-ab82-1db7f60a5212/s3-init/0.log"
Jul 13 20:44:40.295657 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:40.295637    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pkmb9_c09814fd-b698-4cbe-ab82-1db7f60a5212/download-hf-model/0.log"
Jul 13 20:44:40.321999 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:40.321967    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-tnckk_4d64c846-a288-44f3-8857-a483217d8080/seaweedfs/0.log"
Jul 13 20:44:41.692859 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:41.692832    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k5kgn/perf-node-gather-daemonset-4vnsp"
Jul 13 20:44:45.518505 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.518429    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-95p9h_23205ad9-b80f-49b6-9a92-4aeb0654f0e8/kube-multus/0.log"
Jul 13 20:44:45.548013 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.547988    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5crvv_de209486-217b-4fb2-bb81-fb4f185312c1/kube-multus-additional-cni-plugins/0.log"
Jul 13 20:44:45.569482 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.569462    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5crvv_de209486-217b-4fb2-bb81-fb4f185312c1/egress-router-binary-copy/0.log"
Jul 13 20:44:45.589491 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.589464    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5crvv_de209486-217b-4fb2-bb81-fb4f185312c1/cni-plugins/0.log"
Jul 13 20:44:45.609094 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.609073    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5crvv_de209486-217b-4fb2-bb81-fb4f185312c1/bond-cni-plugin/0.log"
Jul 13 20:44:45.629185 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.629158    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5crvv_de209486-217b-4fb2-bb81-fb4f185312c1/routeoverride-cni/0.log"
Jul 13 20:44:45.649639 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.649616    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5crvv_de209486-217b-4fb2-bb81-fb4f185312c1/whereabouts-cni-bincopy/0.log"
Jul 13 20:44:45.670371 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:45.670345    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5crvv_de209486-217b-4fb2-bb81-fb4f185312c1/whereabouts-cni/0.log"
Jul 13 20:44:46.158322 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:46.158290    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6wlbd_bc4b6b4f-5b06-4ace-8cef-2964b68d10fb/network-metrics-daemon/0.log"
Jul 13 20:44:46.179983 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:46.179941    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6wlbd_bc4b6b4f-5b06-4ace-8cef-2964b68d10fb/kube-rbac-proxy/0.log"
Jul 13 20:44:47.419909 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.419880    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-controller/0.log"
Jul 13 20:44:47.440548 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.440522    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/1.log"
Jul 13 20:44:47.440713 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.440567    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovn-acl-logging/0.log"
Jul 13 20:44:47.457416 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.457395    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/kube-rbac-proxy-node/0.log"
Jul 13 20:44:47.477712 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.477690    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/kube-rbac-proxy-ovn-metrics/0.log"
Jul 13 20:44:47.498188 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.498164    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/northd/0.log"
Jul 13 20:44:47.520297 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.520270    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/nbdb/0.log"
Jul 13 20:44:47.541471 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.541452    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/sbdb/0.log"
Jul 13 20:44:47.581041 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:47.581020    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zmz6m_8607e642-2424-4480-bef1-ec3bb89574ed/ovnkube-controller/0.log"
Jul 13 20:44:48.629888 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:48.629858    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xqc68_2771d026-f54c-4c67-a476-d3f5742dab82/network-check-target-container/0.log"
Jul 13 20:44:49.564637 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:49.564610    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-mlkqq_22830676-1f0c-40d7-928d-7809f15bc5db/iptables-alerter/0.log"
Jul 13 20:44:50.217709 ip-10-0-131-42 kubenswrapper[2551]: I0713 20:44:50.217675    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zfhc6_36e04abd-ca7e-44c7-ab7b-441feed30455/tuned/0.log"