Jul 02 14:15:37.640123 ip-10-0-132-82 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 14:15:37.990379 ip-10-0-132-82 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 14:15:37.990379 ip-10-0-132-82 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 14:15:37.990379 ip-10-0-132-82 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 14:15:37.990379 ip-10-0-132-82 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 14:15:37.990379 ip-10-0-132-82 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 14:15:37.991937 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:37.991829    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 14:15:38.000292 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000270    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:15:38.000292 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000288    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:15:38.000292 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:15:38.000292 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000295    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:15:38.000292 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000297    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:15:38.000292 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000300    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:15:38.000292 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000303    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000306    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000308    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000313    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000315    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000318    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000320    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000323    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000326    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000328    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000331    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000334    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000337    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000339    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000342    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000345    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000347    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000350    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000353    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000355    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:15:38.000521 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000358    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000367    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000370    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000374    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000379    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000381    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000384    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000387    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000390    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000393    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000395    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000398    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000400    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000403    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000406    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000409    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000412    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000415    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000417    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000420    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:15:38.001001 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000423    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000425    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000428    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000430    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000433    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000436    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000439    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000441    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000444    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000447    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000449    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000452    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000454    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000457    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000459    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000462    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000464    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000467    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000469    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000471    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:15:38.001501 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000474    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000476    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000479    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000486    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000489    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000492    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000497    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000500    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000503    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000506    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000508    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000511    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000514    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000517    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000520    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000523    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000526    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000529    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000532    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000534    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:15:38.001984 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000537    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000540    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000542    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000545    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000547    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000551    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000555    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000557    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000560    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000562    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000565    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000567    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.000570    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001297    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001305    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001308    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001311    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001314    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001317    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001320    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:15:38.002442 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001322    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001325    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001328    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001331    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001333    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001336    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001338    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001341    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001344    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001346    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001349    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001352    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001354    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001357    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001360    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001362    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001365    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001367    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001370    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001372    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:15:38.002919 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001375    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001378    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001381    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001384    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001387    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001389    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001392    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001394    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001397    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001400    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001403    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001405    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001408    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001410    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001413    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001415    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001418    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001423    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001427    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:15:38.003420 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001430    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001433    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001438    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001441    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001444    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001447    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001450    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001452    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001455    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001458    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001460    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001463    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001465    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001468    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001470    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001473    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001475    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001478    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001480    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001483    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:15:38.003881 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001485    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001487    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001490    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001493    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001495    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001498    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001500    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001511    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001515    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001518    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001521    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001524    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001527    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001529    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001533    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001535    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001540    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001546    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001549    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:15:38.004346 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001552    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001554    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001557    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001559    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001561    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001564    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001566    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001569    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001572    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001574    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001577    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001579    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001581    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001584    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001978    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001983    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001986    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001989    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001992    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001995    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:15:38.004833 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.001998    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002000    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002003    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002006    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002008    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002011    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002013    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002016    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002018    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002021    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002025    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002027    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002030    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002032    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002035    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002037    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002040    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002042    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002045    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:15:38.005282 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002047    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002050    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002052    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002055    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002058    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002062    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002065    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002068    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002071    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002074    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002076    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002079    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002082    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002084    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002087    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002089    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002091    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002094    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002097    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:15:38.005724 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002099    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002101    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002104    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002107    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002109    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002113    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002115    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002117    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002120    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002122    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002125    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002128    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002130    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002132    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002135    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002140    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002143    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002147    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002150    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:15:38.006212 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002153    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002156    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002159    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002161    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002164    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002167    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002169    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002172    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002174    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002177    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002179    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002181    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002184    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002186    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002189    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002191    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002194    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002196    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002199    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002202    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:15:38.006655 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002204    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002208    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002211    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002213    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002216    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002218    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002221    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002223    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002229    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002231    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002234    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002237    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002239    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002242    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002244    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002247    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002316    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002324    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002330    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002334    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002339    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002342    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 14:15:38.007128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002346    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002350    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002354    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002357    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002360    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002363    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002366    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002369    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002372    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002375    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002378    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002381    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002385    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002387    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002390    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002394    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002397    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002401    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002405    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002408    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002411    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002414    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002418    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002421    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 14:15:38.007658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002424    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002427    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002431    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002434    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002437    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002439    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002442    2583 flags.go:64] FLAG: --enable-server="true"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002445    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002450    2583 flags.go:64] FLAG: --event-burst="100"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002453    2583 flags.go:64] FLAG: --event-qps="50"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002459    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002461    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002464    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002468    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002471    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002474    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002478    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002480    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002483    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002487    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002489    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002492    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002495    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002498    2583 flags.go:64] FLAG: --feature-gates=""
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002501    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 14:15:38.008239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002505    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002509    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002512    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002515    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002518    2583 flags.go:64] FLAG: --help="false"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002521    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002524    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002526    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002529    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002532    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002536    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002538    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002541    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002544    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002547    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002550    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002553    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002557    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002560    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002563    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002565    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002568    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002571    2583 flags.go:64] FLAG: --lock-file=""
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002574    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 14:15:38.008890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002576    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002579    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002585    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002588    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002591    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002594    2583 flags.go:64] FLAG: --logging-format="text"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002596    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002600    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002602    2583 flags.go:64] FLAG: --manifest-url=""
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002606    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002611    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002614    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002617    2583 flags.go:64] FLAG: --max-pods="110"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002621    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002623    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002626    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002629    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002632    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002635    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002638    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002645    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002648    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002651    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 14:15:38.009464 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002654    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002657    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002663    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002666    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002669    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002672    2583 flags.go:64] FLAG: --port="10250"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002675    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002678    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05cc8e831cd5e88e8"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002681    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002684    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002687    2583 flags.go:64] FLAG: --register-node="true"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002690    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002693    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002697    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002700    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002702    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002706    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002709    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002712    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002716    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002719    2583 flags.go:64] FLAG: --runonce="false"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002722    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002725    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002728    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002731    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 14:15:38.010029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002733    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002737    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002740    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002743    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002746    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002749    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002752    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002754    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002757    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002761    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002776    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002782    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002785    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002787    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002792    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002794    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002797    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002800    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002803    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002806    2583 flags.go:64] FLAG: --v="2"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002810    2583 flags.go:64] FLAG: --version="false"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002815    2583 flags.go:64] FLAG: --vmodule=""
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002819    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.002822    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002950    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:15:38.010621 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002954    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002958    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002961    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002964    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002967    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002970    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002972    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002975    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002978    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002980    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002983    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002986    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002988    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002991    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002993    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.002996    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003000    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003003    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003005    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003008    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:15:38.011257 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003010    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003012    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003015    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003017    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003020    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003022    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003025    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003028    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003030    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003034    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003036    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003039    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003042    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003046    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003050    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003053    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003056    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003059    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003061    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003064    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:15:38.011726 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003067    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003069    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003072    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003074    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003077    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003079    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003081    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003084    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003089    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003091    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003094    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003097    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003099    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003102    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003105    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003107    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003109    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003112    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003114    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003117    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:15:38.012219 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003120    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003122    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003125    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003128    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003130    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003135    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003137    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003140    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003143    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003145    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003148    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003150    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003153    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003155    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003158    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003160    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003163    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003166    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003168    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:15:38.012696 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003171    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003175    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003177    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003180    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003182    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003186    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003189    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003192    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003195    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003197    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003200    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003202    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003205    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003207    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003210    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003214    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003218    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003221    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:15:38.013171 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.003224    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.003229    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.010156    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.010171    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.010878    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.010886    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010951    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010956    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010960    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010964    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010967    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010970    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010973    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010976    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010979    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:15:38.013609 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010982    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010984    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010987    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010989    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010992    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010995    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.010997    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011000    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011002    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011005    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011008    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011010    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011013    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011015    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011018    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011030    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011033    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011036    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011038    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011041    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:15:38.013986 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011045    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011047    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011050    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011052    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011054    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011057    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011059    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011062    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011066    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011070    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011073    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011076    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011079    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011082    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011085    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011088    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011091    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011094    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011096    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011099    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:15:38.014463 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011101    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011104    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011106    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011109    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011112    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011114    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011117    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011119    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011127    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011130    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011132    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011135    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011138    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011140    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011143    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011145    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011148    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011150    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011153    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011155    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:15:38.014958 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011158    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011160    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011162    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011165    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011167    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011170    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011173    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011176    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011180    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011183    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011185    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011188    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011190    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011192    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011195    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011197    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011200    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011202    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011205    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:15:38.015430 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011207    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011210    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011217    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011220    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011223    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011225    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011229    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011232    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011235    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011238    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011241    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.011246    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011397    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011402    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011405    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 14:15:38.015905 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011408    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011410    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011413    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011416    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011418    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011421    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011424    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011426    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011429    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011432    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011434    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011437    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011439    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011442    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011444    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011447    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011450    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011459    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011462    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011464    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 14:15:38.016261 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011472    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011475    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011477    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011480    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011482    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011485    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011488    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011491    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011495    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011498    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011501    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011504    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011506    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011508    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011511    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011513    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011516    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011519    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011522    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011524    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 14:15:38.016714 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011527    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011529    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011532    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011534    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011537    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011539    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011541    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011544    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011546    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011549    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011552    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011554    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011557    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011565    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011568    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011570    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011573    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011575    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011577    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 14:15:38.017194 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011580    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011582    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011585    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011587    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011590    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011592    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011595    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011597    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011599    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011602    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011604    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011607    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011609    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011612    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011614    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011616    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011619    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011621    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011624    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011627    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 14:15:38.017641 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011629    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011631    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011634    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011636    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011639    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011641    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011643    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011651    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011654    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011657    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011659    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011663    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011666    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011669    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011671    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011674    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 14:15:38.018137 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:38.011676    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 14:15:38.018509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.011681    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 14:15:38.018509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.011802    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 14:15:38.018509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.013905    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 14:15:38.018509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.015375    2583 server.go:1019] "Starting client certificate rotation"
Jul 02 14:15:38.018509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.015506    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 14:15:38.018509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.015539    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 14:15:38.018509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.017798    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 14:15:38.021093 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.021053    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 14:15:38.032683 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.032651    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 02 14:15:38.034753 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.034739    2583 log.go:25] "Validated CRI v1 image API"
Jul 02 14:15:38.035520 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.035506    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 14:15:38.037738 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.037718    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ca92c595-4a4a-451f-91db-e923be95a9b2:/dev/nvme0n1p4 eda2d198-1c29-41e7-9b05-d5ff0f267df3:/dev/nvme0n1p3]
Jul 02 14:15:38.037840 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.037736    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 14:15:38.041887 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.041785    2583 manager.go:216] Machine: {Timestamp:2026-07-02 14:15:38.041304327 +0000 UTC m=+0.296709659 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108428 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fa77171dcf68443db27319ca3b410 SystemUUID:ec2fa771-71dc-f684-43db-27319ca3b410 BootID:34238fc0-3b47-4116-a1b3-ed279b8ec1b4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:88:5b:59:f8:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:88:5b:59:f8:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:8d:9a:9f:8c:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 14:15:38.041887 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.041884    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 14:15:38.041999 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.041983    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 14:15:38.044692 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.044666    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 14:15:38.044868 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.044695    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-82.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 14:15:38.045470 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.045459    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 14:15:38.045501 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.045474    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 14:15:38.045501 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.045481    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 14:15:38.046106 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.046095    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 14:15:38.047033 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.047016    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 14:15:38.047121 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.047041    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 14:15:38.051057 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051045    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 14:15:38.051179 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051169    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 14:15:38.051220 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051199    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 14:15:38.051220 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051208    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 14:15:38.051220 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051218    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 14:15:38.051343 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051229    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 02 14:15:38.051343 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051239    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 14:15:38.051624 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051614    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 14:15:38.051663 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.051636    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 14:15:38.053581 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.053555    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 14:15:38.055663 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.055646    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 14:15:38.055721 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.055678    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 14:15:38.056928 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056913    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 14:15:38.056977 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056937    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 14:15:38.056977 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056943    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 14:15:38.056977 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056949    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 14:15:38.056977 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056957    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 14:15:38.056977 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056962    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 14:15:38.056977 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056971    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 14:15:38.056977 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056976    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 14:15:38.057154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056988    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 14:15:38.057154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.056994    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 14:15:38.057154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.057002    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 14:15:38.057154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.057007    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 14:15:38.057154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.057031    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 14:15:38.057154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.057036    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 14:15:38.060506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.060491    2583 server.go:1268] "Started kubelet"
Jul 02 14:15:38.060610 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.060569    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 14:15:38.064361 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.064189    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 14:15:38.064613 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.064594    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 14:15:38.064730 ip-10-0-132-82 systemd[1]: Started Kubernetes Kubelet.
Jul 02 14:15:38.066445 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.065969    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-82.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 14:15:38.066445 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.066363    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 14:15:38.066620 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.066522    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 14:15:38.066620 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.066584    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 14:15:38.068131 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.068007    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 14:15:38.068946 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.068922    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9xnn8"
Jul 02 14:15:38.069999 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.069199    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-82.ec2.internal.18be7ea24eff7f27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-82.ec2.internal,UID:ip-10-0-132-82.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-82.ec2.internal,},FirstTimestamp:2026-07-02 14:15:38.060463911 +0000 UTC m=+0.315869243,LastTimestamp:2026-07-02 14:15:38.060463911 +0000 UTC m=+0.315869243,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-82.ec2.internal,}"
Jul 02 14:15:38.070674 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.070651    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 14:15:38.073823 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.073808    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 14:15:38.073890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.073807    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 14:15:38.073890 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.073887    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.074424 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074409    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 14:15:38.074535 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074517    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 14:15:38.074623 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074581    2583 factory.go:55] Registering systemd factory
Jul 02 14:15:38.074623 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074593    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 02 14:15:38.074623 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074430    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 14:15:38.074761 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074677    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 14:15:38.074761 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074687    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 14:15:38.074869 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074806    2583 factory.go:153] Registering CRI-O factory
Jul 02 14:15:38.074869 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074818    2583 factory.go:223] Registration of the crio container factory successfully
Jul 02 14:15:38.074931 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074881    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 14:15:38.074931 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074904    2583 factory.go:103] Registering Raw factory
Jul 02 14:15:38.074931 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.074918    2583 manager.go:1196] Started watching for new ooms in manager
Jul 02 14:15:38.075795 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.075780    2583 manager.go:349] Starting recovery of all containers
Jul 02 14:15:38.076103 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.076084    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-82.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 14:15:38.076212 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.076195    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 14:15:38.077563 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.077541    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9xnn8"
Jul 02 14:15:38.086634 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.086617    2583 manager.go:354] Recovery completed
Jul 02 14:15:38.091180 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.091168    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:15:38.092985 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.092966    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:15:38.093063 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.092999    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:15:38.093063 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.093013    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:15:38.093735 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.093725    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 14:15:38.093735 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.093734    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 14:15:38.093849 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.093748    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 14:15:38.093849 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.093804    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:15:38.094481 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.094468    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:15:38.094547 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.094492    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:15:38.094547 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.094501    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:15:38.095857 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.095845    2583 policy_none.go:49] "None policy: Start"
Jul 02 14:15:38.095913 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.095861    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 14:15:38.095913 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.095871    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 14:15:38.097875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.097864    2583 policy_none.go:47] "Start"
Jul 02 14:15:38.134437 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.134329    2583 manager.go:337] "Starting Device Plugin manager"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.134451    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.134461    2583 server.go:87] "Starting device plugin registration server"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.134690    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.134704    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.134841    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.134919    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.134927    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.135425    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.135464    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.141741    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.142809    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.142840    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.142903    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.142939    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 14:15:38.156487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.145426    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:15:38.235141 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.235107    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:15:38.236083 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.236060    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:15:38.236201 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.236098    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:15:38.236201 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.236115    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:15:38.236201 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.236145    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.243350 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.243306    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"]
Jul 02 14:15:38.243406 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.243368    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:15:38.244738 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.244725    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:15:38.244811 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.244752    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:15:38.244811 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.244778    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:15:38.246987 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.246976    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:15:38.247119 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.247197 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247138    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:15:38.247624 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247609    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:15:38.247691 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247640    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:15:38.247691 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247653    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:15:38.247752 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247609    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:15:38.247752 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247720    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:15:38.247752 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.247731    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:15:38.250033 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.250015    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.250120 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.250048    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 14:15:38.250743 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.250725    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 14:15:38.250835 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.250751    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 14:15:38.250835 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.250761    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeHasSufficientPID"
Jul 02 14:15:38.253910 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.253894    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.254162 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.253915    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-82.ec2.internal\": node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.274416 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.274399    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-82.ec2.internal\" not found" node="ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.277893 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.277880    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-82.ec2.internal\" not found" node="ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.302618 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.302592    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.375501 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.375473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a8a2150087e7888b6b104a416079417-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal\" (UID: \"3a8a2150087e7888b6b104a416079417\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.375608 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.375519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a8a2150087e7888b6b104a416079417-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal\" (UID: \"3a8a2150087e7888b6b104a416079417\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.375608 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.375535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6f86f8428c86df58f1aca64ecd7b715-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.ec2.internal\" (UID: \"b6f86f8428c86df58f1aca64ecd7b715\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.403632 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.403611    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.476191 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.476163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a8a2150087e7888b6b104a416079417-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal\" (UID: \"3a8a2150087e7888b6b104a416079417\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.476288 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.476197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a8a2150087e7888b6b104a416079417-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal\" (UID: \"3a8a2150087e7888b6b104a416079417\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.476288 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.476234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6f86f8428c86df58f1aca64ecd7b715-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.ec2.internal\" (UID: \"b6f86f8428c86df58f1aca64ecd7b715\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.476288 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.476269    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a8a2150087e7888b6b104a416079417-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal\" (UID: \"3a8a2150087e7888b6b104a416079417\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.476288 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.476278    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6f86f8428c86df58f1aca64ecd7b715-config\") pod \"kube-apiserver-proxy-ip-10-0-132-82.ec2.internal\" (UID: \"b6f86f8428c86df58f1aca64ecd7b715\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.476461 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.476278    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a8a2150087e7888b6b104a416079417-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal\" (UID: \"3a8a2150087e7888b6b104a416079417\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.504326 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.504247    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.576830 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.576804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.582011 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.581928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:38.604771 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.604745    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.705241 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.705197    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.805801 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.805705    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.906203 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:38.906167    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:38.941506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:38.941485    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:15:39.006957 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:39.006929    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:39.015068 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.015049    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 14:15:39.015203 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.015185    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 14:15:39.015247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.015215    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 02 14:15:39.083404 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.083375    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 14:10:38 +0000 UTC" deadline="2028-02-14 09:17:03.586766329 +0000 UTC"
Jul 02 14:15:39.083404 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.083400    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14203h1m24.503369078s"
Jul 02 14:15:39.107789 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:39.107748    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:39.194906 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:39.194878    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f86f8428c86df58f1aca64ecd7b715.slice/crio-356db0f1d491c3eb5a6b2a52ed2ea89091b0e7f6babef36f4204f89c4638468e WatchSource:0}: Error finding container 356db0f1d491c3eb5a6b2a52ed2ea89091b0e7f6babef36f4204f89c4638468e: Status 404 returned error can't find the container with id 356db0f1d491c3eb5a6b2a52ed2ea89091b0e7f6babef36f4204f89c4638468e
Jul 02 14:15:39.195140 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:39.195117    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8a2150087e7888b6b104a416079417.slice/crio-de681a0bb33bb1041fc49ded1f7c27166972302fcfe2dafe81b1fee8b4ad9aec WatchSource:0}: Error finding container de681a0bb33bb1041fc49ded1f7c27166972302fcfe2dafe81b1fee8b4ad9aec: Status 404 returned error can't find the container with id de681a0bb33bb1041fc49ded1f7c27166972302fcfe2dafe81b1fee8b4ad9aec
Jul 02 14:15:39.199751 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.199738    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:15:39.208459 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:39.208441    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:39.309354 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:39.309315    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-82.ec2.internal\" not found"
Jul 02 14:15:39.329413 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.329338    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:15:39.353975 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.353949    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:15:39.374229 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.374210    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:39.390134 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.390114    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 14:15:39.390969 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.390958    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"
Jul 02 14:15:39.401332 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:39.401319    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 14:15:40.031167 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.031136    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:15:40.052687 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.052650    2583 apiserver.go:52] "Watching apiserver"
Jul 02 14:15:40.062009 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.061985    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 14:15:40.062411 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.062380    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465","openshift-image-registry/node-ca-llvbz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal","openshift-multus/multus-additional-cni-plugins-gg5lm","openshift-multus/multus-mnwkv","openshift-ovn-kubernetes/ovnkube-node-k2cfk","kube-system/global-pull-secret-syncer-88xc5","openshift-cluster-node-tuning-operator/tuned-zfq67","openshift-dns/node-resolver-l5tl5","openshift-multus/network-metrics-daemon-4vgmc","openshift-network-diagnostics/network-check-target-s62pd","openshift-network-operator/iptables-alerter-dmwts","kube-system/konnectivity-agent-dw2fn","kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal"]
Jul 02 14:15:40.064716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.064696    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.064887 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.064861    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:40.066969 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.066947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.069081 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.069065    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.069863 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.069846    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 14:15:40.069863 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.069856    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4sdrg\""
Jul 02 14:15:40.070019 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.069864    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 14:15:40.071197 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.071173    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 14:15:40.071311 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.071290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.071585 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.071468    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 14:15:40.071932 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.071881    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pgl8n\""
Jul 02 14:15:40.074101 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.073498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.074170 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.074100    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 14:15:40.074279 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.074225    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 14:15:40.074337 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.074295    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 14:15:40.074337 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.074307    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 14:15:40.074444 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.074423    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7bhd9\""
Jul 02 14:15:40.074499 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.074485    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 14:15:40.074849 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.074831    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 14:15:40.075902 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.075885    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 14:15:40.075996 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.075981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.076376 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.076154    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-96z5j\""
Jul 02 14:15:40.078325 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.078309    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 14:15:40.079109 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.079085    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 14:15:40.079270 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.079255    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 14:15:40.079349 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.079322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.079546 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.079482    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 14:15:40.080929 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.080904    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 14:15:40.081049 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.081029    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q4kdr\""
Jul 02 14:15:40.081357 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.081340    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 14:15:40.082219 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.082037    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 14:15:40.082219 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.082108    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xlv8s\""
Jul 02 14:15:40.083026 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.082562    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 14:15:40.083906 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.083811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.084841 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.084818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-slash\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.084946 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.084852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-etc-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.084946 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.084885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-system-cni-dir\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.084946 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.084935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-netns\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.085094 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.084968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-ovn\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085094 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.084995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-cni-bin\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085094 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085020    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-cni-netd\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085094 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75a16613-faa9-4c72-ad72-e46f5e821947-ovn-node-metrics-cert\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085094 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fb606c-1df9-4108-95d6-bb407a013248-tmp\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-cni-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-cni-multus\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-multus-certs\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-run-netns\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-log-socket\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-lib-modules\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.085321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cnibin\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-kubelet\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68692d51-6050-407b-8292-af53fb2a0b90-konnectivity-ca\") pod \"konnectivity-agent-dw2fn\" (UID: \"68692d51-6050-407b-8292-af53fb2a0b90\") " pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-socket-dir-parent\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-hostroot\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72mt\" (UniqueName: \"kubernetes.io/projected/75a16613-faa9-4c72-ad72-e46f5e821947-kube-api-access-z72mt\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085470    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-run\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085567    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-env-overrides\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cni-binary-copy\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.085679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysctl-conf\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-systemd\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-sys\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-cnibin\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085854    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-conf-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-systemd-units\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-ovnkube-config\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-k8s-cni-cncf-io\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-cni-bin\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.085992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-etc-kubernetes\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-kubelet\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-var-lib-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aef6263-bcad-4789-89d3-c63178657d4f-dbus\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-systemd\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-ovnkube-script-lib\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086247 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9fb606c-1df9-4108-95d6-bb407a013248-etc-tuned\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzw6\" (UniqueName: \"kubernetes.io/projected/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-kube-api-access-2tzw6\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-node-log\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-os-release\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68692d51-6050-407b-8292-af53fb2a0b90-agent-certs\") pod \"konnectivity-agent-dw2fn\" (UID: \"68692d51-6050-407b-8292-af53fb2a0b90\") " pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-host\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-system-cni-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p68\" (UniqueName: \"kubernetes.io/projected/1627f783-176b-4823-acaa-439926a359dc-kube-api-access-v4p68\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-modprobe-d\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysctl-d\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1627f783-176b-4823-acaa-439926a359dc-cni-binary-copy\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-os-release\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1627f783-176b-4823-acaa-439926a359dc-multus-daemon-config\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aef6263-bcad-4789-89d3-c63178657d4f-kubelet-config\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-var-lib-kubelet\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.086875 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8248\" (UniqueName: \"kubernetes.io/projected/c9fb606c-1df9-4108-95d6-bb407a013248-kube-api-access-h8248\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysconfig\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-kubernetes\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086926    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086959    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.086998    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.087185    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5g7b4\""
Jul 02 14:15:40.087575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.087213    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.089558 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.089540    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 14:15:40.089653 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.089618    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 14:15:40.089711 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.089654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:40.089813 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.089719    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:40.090092 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.090072    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dqqvk\""
Jul 02 14:15:40.090163 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.090143    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 14:15:40.092328 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.092310    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:40.092412 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.092391    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:40.094898 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.094631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.098294 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.098257    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 14:15:40.098398 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.098322    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 14:15:40.098398 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.098356    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7947n\""
Jul 02 14:15:40.098575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.098498    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 14:15:40.108647 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.108629    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qw7b7"
Jul 02 14:15:40.116925 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.116908    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qw7b7"
Jul 02 14:15:40.147652 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.147601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal" event={"ID":"b6f86f8428c86df58f1aca64ecd7b715","Type":"ContainerStarted","Data":"356db0f1d491c3eb5a6b2a52ed2ea89091b0e7f6babef36f4204f89c4638468e"}
Jul 02 14:15:40.148716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.148694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal" event={"ID":"3a8a2150087e7888b6b104a416079417","Type":"ContainerStarted","Data":"de681a0bb33bb1041fc49ded1f7c27166972302fcfe2dafe81b1fee8b4ad9aec"}
Jul 02 14:15:40.176338 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.176315    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 14:15:40.187828 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.187805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-hostroot\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.187959 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.187833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72mt\" (UniqueName: \"kubernetes.io/projected/75a16613-faa9-4c72-ad72-e46f5e821947-kube-api-access-z72mt\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.187959 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.187858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-run\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.187959 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.187883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.187959 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.187906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.187959 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.187929    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-hostroot\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.187979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-env-overrides\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cni-binary-copy\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188046    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188059    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysctl-conf\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-systemd\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-sys\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-cnibin\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-conf-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-systemd-units\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188234    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysctl-conf\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188291    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-cnibin\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188300    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-conf-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188312    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-sys\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-ovnkube-config\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188341    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-systemd-units\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188392    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-systemd\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-k8s-cni-cncf-io\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-cni-bin\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188481    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188478    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-env-overrides\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-etc-kubernetes\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188537    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-cni-bin\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188536    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-k8s-cni-cncf-io\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-kubelet\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-var-lib-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.188716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aef6263-bcad-4789-89d3-c63178657d4f-dbus\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188607    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-etc-kubernetes\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188653    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cni-binary-copy\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188654    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-var-lib-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c88b0850-2950-401e-8569-fe7232e027a0-host-slash\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188714    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-kubelet\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-systemd\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-ovnkube-script-lib\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.188779    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9fb606c-1df9-4108-95d6-bb407a013248-etc-tuned\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188815    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aef6263-bcad-4789-89d3-c63178657d4f-dbus\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzw6\" (UniqueName: \"kubernetes.io/projected/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-kube-api-access-2tzw6\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.188860    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret podName:2aef6263-bcad-4789-89d3-c63178657d4f nodeName:}" failed. No retries permitted until 2026-07-02 14:15:40.688822498 +0000 UTC m=+2.944227821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret") pod "global-pull-secret-syncer-88xc5" (UID: "2aef6263-bcad-4789-89d3-c63178657d4f") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188919    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-systemd\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188981    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-ovnkube-config\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.188984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-registration-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.189556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189032    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-device-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292e4555-5517-496c-b83a-b1e4090e0579-tmp-dir\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhxr\" (UniqueName: \"kubernetes.io/projected/36221c70-cb82-48b5-8d8c-d6babeccf3b5-kube-api-access-hqhxr\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-node-log\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-socket-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189121    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189149    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbgg\" (UniqueName: \"kubernetes.io/projected/292e4555-5517-496c-b83a-b1e4090e0579-kube-api-access-8dbgg\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36221c70-cb82-48b5-8d8c-d6babeccf3b5-host\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189205    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-node-log\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-os-release\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189275    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75a16613-faa9-4c72-ad72-e46f5e821947-ovnkube-script-lib\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68692d51-6050-407b-8292-af53fb2a0b90-agent-certs\") pod \"konnectivity-agent-dw2fn\" (UID: \"68692d51-6050-407b-8292-af53fb2a0b90\") " pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-host\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189345    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-os-release\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djk2\" (UniqueName: \"kubernetes.io/projected/cc2a1dba-58ab-488f-a113-e51af9710db3-kube-api-access-4djk2\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.190231 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189387    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-host\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-system-cni-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189404    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-run\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p68\" (UniqueName: \"kubernetes.io/projected/1627f783-176b-4823-acaa-439926a359dc-kube-api-access-v4p68\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189476    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-system-cni-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-modprobe-d\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysctl-d\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189538    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxq68\" (UniqueName: \"kubernetes.io/projected/c88b0850-2950-401e-8569-fe7232e027a0-kube-api-access-bxq68\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36221c70-cb82-48b5-8d8c-d6babeccf3b5-serviceca\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1627f783-176b-4823-acaa-439926a359dc-cni-binary-copy\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189583    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-modprobe-d\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-os-release\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189623    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysctl-d\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8zc\" (UniqueName: \"kubernetes.io/projected/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-kube-api-access-6x8zc\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189680    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-os-release\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.191038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1627f783-176b-4823-acaa-439926a359dc-multus-daemon-config\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aef6263-bcad-4789-89d3-c63178657d4f-kubelet-config\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-var-lib-kubelet\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8248\" (UniqueName: \"kubernetes.io/projected/c9fb606c-1df9-4108-95d6-bb407a013248-kube-api-access-h8248\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189839    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/292e4555-5517-496c-b83a-b1e4090e0579-hosts-file\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysconfig\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-kubernetes\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-slash\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189915    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aef6263-bcad-4789-89d3-c63178657d4f-kubelet-config\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-etc-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-system-cni-dir\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189977    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-sysconfig\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.189992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-netns\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-ovn\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-cni-bin\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-cni-netd\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.191799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75a16613-faa9-4c72-ad72-e46f5e821947-ovn-node-metrics-cert\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fb606c-1df9-4108-95d6-bb407a013248-tmp\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-sys-fs\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-cni-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190181    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1627f783-176b-4823-acaa-439926a359dc-cni-binary-copy\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-cni-multus\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-multus-certs\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190234    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-ovn\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-run-netns\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190305    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-slash\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-log-socket\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190340    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-etc-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-lib-modules\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190357    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1627f783-176b-4823-acaa-439926a359dc-multus-daemon-config\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190376    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-system-cni-dir\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cnibin\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-kubelet\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.192599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68692d51-6050-407b-8292-af53fb2a0b90-konnectivity-ca\") pod \"konnectivity-agent-dw2fn\" (UID: \"68692d51-6050-407b-8292-af53fb2a0b90\") " pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190455    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-netns\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190458    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-etc-kubernetes\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190272    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-var-lib-kubelet\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c88b0850-2950-401e-8569-fe7232e027a0-iptables-alerter-script\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190498    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-run-netns\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190519    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-cnibin\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190533    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-run-openvswitch\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190550    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-kubelet\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190418    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-var-lib-cni-multus\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190576    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-log-socket\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190586    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-host-run-multus-certs\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-socket-dir-parent\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190654    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9fb606c-1df9-4108-95d6-bb407a013248-lib-modules\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190730    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-cni-dir\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190736    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1627f783-176b-4823-acaa-439926a359dc-multus-socket-dir-parent\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190787    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-cni-netd\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.193261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.190844    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-cni-bin\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.193925 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.191109    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/68692d51-6050-407b-8292-af53fb2a0b90-konnectivity-ca\") pod \"konnectivity-agent-dw2fn\" (UID: \"68692d51-6050-407b-8292-af53fb2a0b90\") " pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.193925 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.191221    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75a16613-faa9-4c72-ad72-e46f5e821947-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.193925 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.192870    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fb606c-1df9-4108-95d6-bb407a013248-tmp\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.193925 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.193013    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9fb606c-1df9-4108-95d6-bb407a013248-etc-tuned\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.193925 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.193182    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/68692d51-6050-407b-8292-af53fb2a0b90-agent-certs\") pod \"konnectivity-agent-dw2fn\" (UID: \"68692d51-6050-407b-8292-af53fb2a0b90\") " pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.193925 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.193298    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75a16613-faa9-4c72-ad72-e46f5e821947-ovn-node-metrics-cert\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.200024 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.200000    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzw6\" (UniqueName: \"kubernetes.io/projected/bc51f60a-9cbb-4c55-9df2-faaa95d7822d-kube-api-access-2tzw6\") pod \"multus-additional-cni-plugins-gg5lm\" (UID: \"bc51f60a-9cbb-4c55-9df2-faaa95d7822d\") " pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.200979 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.200932    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72mt\" (UniqueName: \"kubernetes.io/projected/75a16613-faa9-4c72-ad72-e46f5e821947-kube-api-access-z72mt\") pod \"ovnkube-node-k2cfk\" (UID: \"75a16613-faa9-4c72-ad72-e46f5e821947\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.201567 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.201542    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8248\" (UniqueName: \"kubernetes.io/projected/c9fb606c-1df9-4108-95d6-bb407a013248-kube-api-access-h8248\") pod \"tuned-zfq67\" (UID: \"c9fb606c-1df9-4108-95d6-bb407a013248\") " pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.201897 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.201881    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p68\" (UniqueName: \"kubernetes.io/projected/1627f783-176b-4823-acaa-439926a359dc-kube-api-access-v4p68\") pod \"multus-mnwkv\" (UID: \"1627f783-176b-4823-acaa-439926a359dc\") " pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.291557 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c88b0850-2950-401e-8569-fe7232e027a0-iptables-alerter-script\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.291712 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.291712 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c88b0850-2950-401e-8569-fe7232e027a0-host-slash\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.291712 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-registration-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.291712 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-device-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.291712 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292e4555-5517-496c-b83a-b1e4090e0579-tmp-dir\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.291712 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291698    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c88b0850-2950-401e-8569-fe7232e027a0-host-slash\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhxr\" (UniqueName: \"kubernetes.io/projected/36221c70-cb82-48b5-8d8c-d6babeccf3b5-kube-api-access-hqhxr\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291743    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-registration-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-socket-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbgg\" (UniqueName: \"kubernetes.io/projected/292e4555-5517-496c-b83a-b1e4090e0579-kube-api-access-8dbgg\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291808    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-device-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291743    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-etc-selinux\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36221c70-cb82-48b5-8d8c-d6babeccf3b5-host\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djk2\" (UniqueName: \"kubernetes.io/projected/cc2a1dba-58ab-488f-a113-e51af9710db3-kube-api-access-4djk2\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291925    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/36221c70-cb82-48b5-8d8c-d6babeccf3b5-host\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxq68\" (UniqueName: \"kubernetes.io/projected/c88b0850-2950-401e-8569-fe7232e027a0-kube-api-access-bxq68\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36221c70-cb82-48b5-8d8c-d6babeccf3b5-serviceca\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292042 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8zc\" (UniqueName: \"kubernetes.io/projected/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-kube-api-access-6x8zc\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/292e4555-5517-496c-b83a-b1e4090e0579-hosts-file\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292036    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292e4555-5517-496c-b83a-b1e4090e0579-tmp-dir\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.291892    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-socket-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-sys-fs\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292135    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292169    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc2a1dba-58ab-488f-a113-e51af9710db3-sys-fs\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292172    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c88b0850-2950-401e-8569-fe7232e027a0-iptables-alerter-script\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.292158    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292231    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/292e4555-5517-496c-b83a-b1e4090e0579-hosts-file\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.292256    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs podName:46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:40.792237142 +0000 UTC m=+3.047642465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs") pod "network-metrics-daemon-4vgmc" (UID: "46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:40.292586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.292501    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/36221c70-cb82-48b5-8d8c-d6babeccf3b5-serviceca\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.301693 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.301670    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:15:40.301841 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.301698    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:15:40.301841 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.301711    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wxwbl for pod openshift-network-diagnostics/network-check-target-s62pd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:40.301841 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.301781    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl podName:feefb507-74b8-4f61-b823-359ea3ba6462 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:40.80175062 +0000 UTC m=+3.057155955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wxwbl" (UniqueName: "kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl") pod "network-check-target-s62pd" (UID: "feefb507-74b8-4f61-b823-359ea3ba6462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:40.302179 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.302137    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxq68\" (UniqueName: \"kubernetes.io/projected/c88b0850-2950-401e-8569-fe7232e027a0-kube-api-access-bxq68\") pod \"iptables-alerter-dmwts\" (UID: \"c88b0850-2950-401e-8569-fe7232e027a0\") " pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.302334 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.302310    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhxr\" (UniqueName: \"kubernetes.io/projected/36221c70-cb82-48b5-8d8c-d6babeccf3b5-kube-api-access-hqhxr\") pod \"node-ca-llvbz\" (UID: \"36221c70-cb82-48b5-8d8c-d6babeccf3b5\") " pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.302532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.302511    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djk2\" (UniqueName: \"kubernetes.io/projected/cc2a1dba-58ab-488f-a113-e51af9710db3-kube-api-access-4djk2\") pod \"aws-ebs-csi-driver-node-5z465\" (UID: \"cc2a1dba-58ab-488f-a113-e51af9710db3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.304168 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.304148    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbgg\" (UniqueName: \"kubernetes.io/projected/292e4555-5517-496c-b83a-b1e4090e0579-kube-api-access-8dbgg\") pod \"node-resolver-l5tl5\" (UID: \"292e4555-5517-496c-b83a-b1e4090e0579\") " pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.304420 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.304402    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8zc\" (UniqueName: \"kubernetes.io/projected/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-kube-api-access-6x8zc\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:40.377877 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.377838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:15:40.387190 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.387162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zfq67"
Jul 02 14:15:40.396188 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.396160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gg5lm"
Jul 02 14:15:40.399778 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.399741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mnwkv"
Jul 02 14:15:40.407338 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.407318    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:15:40.412989 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.412972    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5tl5"
Jul 02 14:15:40.418540 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.418524    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465"
Jul 02 14:15:40.426046 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.426019    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-llvbz"
Jul 02 14:15:40.432653 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.432637    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dmwts"
Jul 02 14:15:40.695756 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.695688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:40.695921 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.695824    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:40.695921 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.695895    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret podName:2aef6263-bcad-4789-89d3-c63178657d4f nodeName:}" failed. No retries permitted until 2026-07-02 14:15:41.695878354 +0000 UTC m=+3.951283674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret") pod "global-pull-secret-syncer-88xc5" (UID: "2aef6263-bcad-4789-89d3-c63178657d4f") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:40.796927 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.796897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:40.797085 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.797022    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:40.797132 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.797085    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs podName:46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:41.797066497 +0000 UTC m=+4.052471817 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs") pod "network-metrics-daemon-4vgmc" (UID: "46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:40.897277 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:40.897241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:40.897441 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.897407    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:15:40.897441 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.897428    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:15:40.897441 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.897440    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wxwbl for pod openshift-network-diagnostics/network-check-target-s62pd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:40.897576 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:40.897490    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl podName:feefb507-74b8-4f61-b823-359ea3ba6462 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:41.897477349 +0000 UTC m=+4.152882672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxwbl" (UniqueName: "kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl") pod "network-check-target-s62pd" (UID: "feefb507-74b8-4f61-b823-359ea3ba6462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:41.084535 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.084296    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc51f60a_9cbb_4c55_9df2_faaa95d7822d.slice/crio-969d82fddfda4389f30790423610d47416f076d6ac2f966f0263be56cadd0f07 WatchSource:0}: Error finding container 969d82fddfda4389f30790423610d47416f076d6ac2f966f0263be56cadd0f07: Status 404 returned error can't find the container with id 969d82fddfda4389f30790423610d47416f076d6ac2f966f0263be56cadd0f07
Jul 02 14:15:41.086155 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.086135    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1627f783_176b_4823_acaa_439926a359dc.slice/crio-53a82b13643562d455655963775c9e3cfe11868c4d8c4cec63b90546fbc87e5e WatchSource:0}: Error finding container 53a82b13643562d455655963775c9e3cfe11868c4d8c4cec63b90546fbc87e5e: Status 404 returned error can't find the container with id 53a82b13643562d455655963775c9e3cfe11868c4d8c4cec63b90546fbc87e5e
Jul 02 14:15:41.086627 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.086605    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292e4555_5517_496c_b83a_b1e4090e0579.slice/crio-fb95dbb95412bd63b1b6d5d2bb8e7eb9d1e7c6a13ec95762e0f21bfdcc1ac59a WatchSource:0}: Error finding container fb95dbb95412bd63b1b6d5d2bb8e7eb9d1e7c6a13ec95762e0f21bfdcc1ac59a: Status 404 returned error can't find the container with id fb95dbb95412bd63b1b6d5d2bb8e7eb9d1e7c6a13ec95762e0f21bfdcc1ac59a
Jul 02 14:15:41.089451 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.089426    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88b0850_2950_401e_8569_fe7232e027a0.slice/crio-a68604c4cef973339139744bb46d251d06a7e1f3dc1fdfe37e6ab3d6b7603c7f WatchSource:0}: Error finding container a68604c4cef973339139744bb46d251d06a7e1f3dc1fdfe37e6ab3d6b7603c7f: Status 404 returned error can't find the container with id a68604c4cef973339139744bb46d251d06a7e1f3dc1fdfe37e6ab3d6b7603c7f
Jul 02 14:15:41.112554 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.112527    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36221c70_cb82_48b5_8d8c_d6babeccf3b5.slice/crio-086f6164159892d21b4f21c99faf6d0052bf4a9043939ee3011393655ba3d0ce WatchSource:0}: Error finding container 086f6164159892d21b4f21c99faf6d0052bf4a9043939ee3011393655ba3d0ce: Status 404 returned error can't find the container with id 086f6164159892d21b4f21c99faf6d0052bf4a9043939ee3011393655ba3d0ce
Jul 02 14:15:41.113224 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.113203    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2a1dba_58ab_488f_a113_e51af9710db3.slice/crio-e13f0658474e38d78cf2ff173153922e4570d4c0f54daa7576a4fe2cfa8ece7a WatchSource:0}: Error finding container e13f0658474e38d78cf2ff173153922e4570d4c0f54daa7576a4fe2cfa8ece7a: Status 404 returned error can't find the container with id e13f0658474e38d78cf2ff173153922e4570d4c0f54daa7576a4fe2cfa8ece7a
Jul 02 14:15:41.114071 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.114053    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a16613_faa9_4c72_ad72_e46f5e821947.slice/crio-4a3472dc1b86f6940dcf2ebd1ff0717e8dd9bff1a773d7fdbb58cb61887b8854 WatchSource:0}: Error finding container 4a3472dc1b86f6940dcf2ebd1ff0717e8dd9bff1a773d7fdbb58cb61887b8854: Status 404 returned error can't find the container with id 4a3472dc1b86f6940dcf2ebd1ff0717e8dd9bff1a773d7fdbb58cb61887b8854
Jul 02 14:15:41.116120 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.115810    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fb606c_1df9_4108_95d6_bb407a013248.slice/crio-1c6ee4211c371d3814b346da294044331e14d3cf6623dc309c8ee221a17697b7 WatchSource:0}: Error finding container 1c6ee4211c371d3814b346da294044331e14d3cf6623dc309c8ee221a17697b7: Status 404 returned error can't find the container with id 1c6ee4211c371d3814b346da294044331e14d3cf6623dc309c8ee221a17697b7
Jul 02 14:15:41.116668 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:15:41.116607    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68692d51_6050_407b_8292_af53fb2a0b90.slice/crio-4b4f9a51d97037023e8de63f44781bf5f6acfc2904fe4f0d4d05d3ccbfcee829 WatchSource:0}: Error finding container 4b4f9a51d97037023e8de63f44781bf5f6acfc2904fe4f0d4d05d3ccbfcee829: Status 404 returned error can't find the container with id 4b4f9a51d97037023e8de63f44781bf5f6acfc2904fe4f0d4d05d3ccbfcee829
Jul 02 14:15:41.119509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.119465    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 14:10:40 +0000 UTC" deadline="2028-03-10 14:32:36.325292074 +0000 UTC"
Jul 02 14:15:41.119509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.119505    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14808h16m55.20579074s"
Jul 02 14:15:41.152697 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.152671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dw2fn" event={"ID":"68692d51-6050-407b-8292-af53fb2a0b90","Type":"ContainerStarted","Data":"4b4f9a51d97037023e8de63f44781bf5f6acfc2904fe4f0d4d05d3ccbfcee829"}
Jul 02 14:15:41.153816 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.153791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zfq67" event={"ID":"c9fb606c-1df9-4108-95d6-bb407a013248","Type":"ContainerStarted","Data":"1c6ee4211c371d3814b346da294044331e14d3cf6623dc309c8ee221a17697b7"}
Jul 02 14:15:41.154846 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.154826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465" event={"ID":"cc2a1dba-58ab-488f-a113-e51af9710db3","Type":"ContainerStarted","Data":"e13f0658474e38d78cf2ff173153922e4570d4c0f54daa7576a4fe2cfa8ece7a"}
Jul 02 14:15:41.155810 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.155787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llvbz" event={"ID":"36221c70-cb82-48b5-8d8c-d6babeccf3b5","Type":"ContainerStarted","Data":"086f6164159892d21b4f21c99faf6d0052bf4a9043939ee3011393655ba3d0ce"}
Jul 02 14:15:41.156813 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.156788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mnwkv" event={"ID":"1627f783-176b-4823-acaa-439926a359dc","Type":"ContainerStarted","Data":"53a82b13643562d455655963775c9e3cfe11868c4d8c4cec63b90546fbc87e5e"}
Jul 02 14:15:41.161874 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.161853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerStarted","Data":"969d82fddfda4389f30790423610d47416f076d6ac2f966f0263be56cadd0f07"}
Jul 02 14:15:41.164077 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.164058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"4a3472dc1b86f6940dcf2ebd1ff0717e8dd9bff1a773d7fdbb58cb61887b8854"}
Jul 02 14:15:41.164978 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.164960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dmwts" event={"ID":"c88b0850-2950-401e-8569-fe7232e027a0","Type":"ContainerStarted","Data":"a68604c4cef973339139744bb46d251d06a7e1f3dc1fdfe37e6ab3d6b7603c7f"}
Jul 02 14:15:41.165820 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.165801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5tl5" event={"ID":"292e4555-5517-496c-b83a-b1e4090e0579","Type":"ContainerStarted","Data":"fb95dbb95412bd63b1b6d5d2bb8e7eb9d1e7c6a13ec95762e0f21bfdcc1ac59a"}
Jul 02 14:15:41.704649 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.704374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:41.704649 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.704604    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:41.704918 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.704677    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret podName:2aef6263-bcad-4789-89d3-c63178657d4f nodeName:}" failed. No retries permitted until 2026-07-02 14:15:43.704657334 +0000 UTC m=+5.960062667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret") pod "global-pull-secret-syncer-88xc5" (UID: "2aef6263-bcad-4789-89d3-c63178657d4f") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:41.805300 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.805264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:41.805460 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.805429    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:41.805555 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.805491    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs podName:46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:43.805474094 +0000 UTC m=+6.060879434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs") pod "network-metrics-daemon-4vgmc" (UID: "46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:41.829023 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.828994    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 14:15:41.906146 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:41.906109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:41.906364 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.906303    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:15:41.906364 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.906326    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:15:41.906364 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.906339    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wxwbl for pod openshift-network-diagnostics/network-check-target-s62pd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:41.906581 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:41.906397    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl podName:feefb507-74b8-4f61-b823-359ea3ba6462 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:43.906378791 +0000 UTC m=+6.161784113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxwbl" (UniqueName: "kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl") pod "network-check-target-s62pd" (UID: "feefb507-74b8-4f61-b823-359ea3ba6462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:42.120562 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.120479    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 14:10:40 +0000 UTC" deadline="2028-03-17 05:30:26.624461584 +0000 UTC"
Jul 02 14:15:42.120562 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.120514    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14967h14m44.503952144s"
Jul 02 14:15:42.145269 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.144904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:42.145269 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.144942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:42.145269 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:42.145028    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:42.145269 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:42.145092    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:42.145269 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.145151    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:42.145269 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:42.145228    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:42.185901 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.185865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal" event={"ID":"b6f86f8428c86df58f1aca64ecd7b715","Type":"ContainerStarted","Data":"fe2106d3bb4bbe39a76d8226fbc977b32d09980d2a40cbed6093f34b492ca438"}
Jul 02 14:15:42.214344 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.214303    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a8a2150087e7888b6b104a416079417" containerID="dfe96cc95e100399558956a55afbad3c87f2bd530d5e2bcddd91a583258e8fd1" exitCode=0
Jul 02 14:15:42.214530 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.214361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal" event={"ID":"3a8a2150087e7888b6b104a416079417","Type":"ContainerDied","Data":"dfe96cc95e100399558956a55afbad3c87f2bd530d5e2bcddd91a583258e8fd1"}
Jul 02 14:15:42.232098 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:42.232040    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-82.ec2.internal" podStartSLOduration=3.232017556 podStartE2EDuration="3.232017556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:15:42.202153894 +0000 UTC m=+4.457559237" watchObservedRunningTime="2026-07-02 14:15:42.232017556 +0000 UTC m=+4.487422902"
Jul 02 14:15:43.243215 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:43.243182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal" event={"ID":"3a8a2150087e7888b6b104a416079417","Type":"ContainerStarted","Data":"2decc992aba1cd0083491879565dd3533331ba6c1519f2b05fd09feda7d75757"}
Jul 02 14:15:43.721946 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:43.721255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:43.721946 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.721452    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:43.721946 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.721515    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret podName:2aef6263-bcad-4789-89d3-c63178657d4f nodeName:}" failed. No retries permitted until 2026-07-02 14:15:47.721496739 +0000 UTC m=+9.976902064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret") pod "global-pull-secret-syncer-88xc5" (UID: "2aef6263-bcad-4789-89d3-c63178657d4f") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:43.822083 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:43.822046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:43.822651 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.822243    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:43.822651 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.822315    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs podName:46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:47.822295313 +0000 UTC m=+10.077700650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs") pod "network-metrics-daemon-4vgmc" (UID: "46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:43.923343 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:43.922664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:43.923343 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.922901    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:15:43.923343 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.922924    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:15:43.923343 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.922937    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wxwbl for pod openshift-network-diagnostics/network-check-target-s62pd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:43.923343 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:43.923003    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl podName:feefb507-74b8-4f61-b823-359ea3ba6462 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:47.922977553 +0000 UTC m=+10.178382887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxwbl" (UniqueName: "kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl") pod "network-check-target-s62pd" (UID: "feefb507-74b8-4f61-b823-359ea3ba6462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:44.144122 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:44.144040    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:44.144122 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:44.144120    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:44.144334 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:44.144119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:44.144334 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:44.144255    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:44.144446 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:44.144352    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:44.144446 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:44.144428    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:46.144128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:46.143399    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:46.144128 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:46.143539    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:46.144128 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:46.143988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:46.144128 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:46.144085    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:46.144732 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:46.144703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:46.144839 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:46.144813    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:47.756417 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:47.756381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:47.756924 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.756545    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:47.756924 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.756621    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret podName:2aef6263-bcad-4789-89d3-c63178657d4f nodeName:}" failed. No retries permitted until 2026-07-02 14:15:55.756598766 +0000 UTC m=+18.012004100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret") pod "global-pull-secret-syncer-88xc5" (UID: "2aef6263-bcad-4789-89d3-c63178657d4f") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:47.857383 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:47.857354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:47.857553 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.857470    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:47.857553 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.857515    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs podName:46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:55.857502521 +0000 UTC m=+18.112907840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs") pod "network-metrics-daemon-4vgmc" (UID: "46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:47.958604 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:47.958568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:47.958809 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.958782    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:15:47.958809 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.958814    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:15:47.958980 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.958827    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wxwbl for pod openshift-network-diagnostics/network-check-target-s62pd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:47.958980 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:47.958894    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl podName:feefb507-74b8-4f61-b823-359ea3ba6462 nodeName:}" failed. No retries permitted until 2026-07-02 14:15:55.958874364 +0000 UTC m=+18.214279700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxwbl" (UniqueName: "kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl") pod "network-check-target-s62pd" (UID: "feefb507-74b8-4f61-b823-359ea3ba6462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:48.148787 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:48.148696    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:48.148939 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:48.148824    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:48.149058 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:48.149022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:48.149185 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:48.149123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:48.149185 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:48.149173    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:48.149275 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:48.149200    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:50.143463 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:50.143430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:50.143463 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:50.143448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:50.143978 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:50.143542    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:50.143978 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:50.143619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:50.143978 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:50.143683    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:50.143978 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:50.143609    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:52.143347 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:52.143312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:52.143837 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:52.143366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:52.143837 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:52.143453    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:52.143837 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:52.143526    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:52.143837 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:52.143580    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:52.143837 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:52.143666    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:54.143573 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:54.143535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:54.144031 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:54.143536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:54.144031 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:54.143678    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:54.144031 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:54.143743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:54.144031 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:54.143859    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:54.144031 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:54.143934    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:55.815158 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:55.815120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:55.815693 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:55.815278    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:55.815693 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:55.815362    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret podName:2aef6263-bcad-4789-89d3-c63178657d4f nodeName:}" failed. No retries permitted until 2026-07-02 14:16:11.815342913 +0000 UTC m=+34.070748236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret") pod "global-pull-secret-syncer-88xc5" (UID: "2aef6263-bcad-4789-89d3-c63178657d4f") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:15:55.915784 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:55.915736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:55.915937 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:55.915886    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:55.915996 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:55.915948    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs podName:46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:11.915931402 +0000 UTC m=+34.171336728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs") pod "network-metrics-daemon-4vgmc" (UID: "46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:15:56.016974 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:56.016940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:56.017147 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:56.017126    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:15:56.017193 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:56.017154    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:15:56.017193 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:56.017167    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wxwbl for pod openshift-network-diagnostics/network-check-target-s62pd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:56.017279 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:56.017224    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl podName:feefb507-74b8-4f61-b823-359ea3ba6462 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:12.017207758 +0000 UTC m=+34.272613098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxwbl" (UniqueName: "kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl") pod "network-check-target-s62pd" (UID: "feefb507-74b8-4f61-b823-359ea3ba6462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:15:56.143682 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:56.143596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:56.143682 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:56.143620    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:56.143920 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:56.143596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:56.143920 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:56.143729    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:56.143920 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:56.143839    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:15:56.143920 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:56.143901    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:58.144316 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:58.144275    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:15:58.144794 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:58.144371    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:15:58.144794 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:58.144393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:15:58.144794 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:15:58.144416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:15:58.144794 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:58.144571    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:15:58.144794 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:15:58.144655    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:00.144894 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.144589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:00.144894 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:00.144698    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:00.144894 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.144780    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:00.144894 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:00.144874    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:00.145398 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.145055    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:00.145398 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:00.145165    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:00.281220 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.281184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465" event={"ID":"cc2a1dba-58ab-488f-a113-e51af9710db3","Type":"ContainerStarted","Data":"2f3ef44c058948a3a1247fcb260d4f80b049ccd8d0106746e476e2858845c4d1"}
Jul 02 14:16:00.282740 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.282700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-llvbz" event={"ID":"36221c70-cb82-48b5-8d8c-d6babeccf3b5","Type":"ContainerStarted","Data":"ddfbef9f880622f473948de47d34107ebf2d7a04a599da87bad42fa5f321fe01"}
Jul 02 14:16:00.284340 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.284315    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mnwkv" event={"ID":"1627f783-176b-4823-acaa-439926a359dc","Type":"ContainerStarted","Data":"2cfd6b2b5d91da2f82358139b1af3e9bbd136bc587fa4c171730b2388f391fa3"}
Jul 02 14:16:00.285649 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.285626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerStarted","Data":"c4f40105796f944b5f10e58f465e42366a70ec8dd629ffdc8b55457b1b7dc553"}
Jul 02 14:16:00.286731 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.286711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5tl5" event={"ID":"292e4555-5517-496c-b83a-b1e4090e0579","Type":"ContainerStarted","Data":"5fa89400401e0024743f9cea6ebce531756eee4b66c32f317d0a88add75f3bea"}
Jul 02 14:16:00.288145 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.288125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dw2fn" event={"ID":"68692d51-6050-407b-8292-af53fb2a0b90","Type":"ContainerStarted","Data":"71489fab1873a5091a0f99506614d43aa62be4acd871e0583e3c30cb17c2ef6a"}
Jul 02 14:16:00.289490 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.289472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zfq67" event={"ID":"c9fb606c-1df9-4108-95d6-bb407a013248","Type":"ContainerStarted","Data":"5636d32a7729cc40841d363eea6eba689929978ded0d45ffebaa35f5ba2ca651"}
Jul 02 14:16:00.298961 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.298920    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-llvbz" podStartSLOduration=3.739824446 podStartE2EDuration="22.298909502s" totalImagesPullingTime="18.559085056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:00.298875471 +0000 UTC m=+22.554280812" watchObservedRunningTime="2026-07-02 14:16:00.298909502 +0000 UTC m=+22.554314842"
Jul 02 14:16:00.299081 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.299006    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-82.ec2.internal" podStartSLOduration=21.298998619 podStartE2EDuration="21.298998619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:15:43.263668894 +0000 UTC m=+5.519074235" watchObservedRunningTime="2026-07-02 14:16:00.298998619 +0000 UTC m=+22.554403962"
Jul 02 14:16:00.314294 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.314258    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dw2fn" podStartSLOduration=3.434917586 podStartE2EDuration="22.314247341s" totalImagesPullingTime="18.879329755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:00.313745787 +0000 UTC m=+22.569151129" watchObservedRunningTime="2026-07-02 14:16:00.314247341 +0000 UTC m=+22.569652681"
Jul 02 14:16:00.369646 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.369604    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mnwkv" podStartSLOduration=3.44066277 podStartE2EDuration="22.369590147s" totalImagesPullingTime="18.928927377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:00.352488713 +0000 UTC m=+22.607894065" watchObservedRunningTime="2026-07-02 14:16:00.369590147 +0000 UTC m=+22.624995488"
Jul 02 14:16:00.369865 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.369844    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zfq67" podStartSLOduration=3.471841241 podStartE2EDuration="22.369838196s" totalImagesPullingTime="18.897996955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:00.369563473 +0000 UTC m=+22.624968804" watchObservedRunningTime="2026-07-02 14:16:00.369838196 +0000 UTC m=+22.625243536"
Jul 02 14:16:00.386239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:00.386199    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l5tl5" podStartSLOduration=3.498440409 podStartE2EDuration="22.386186138s" totalImagesPullingTime="18.887745729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:00.385880149 +0000 UTC m=+22.641285490" watchObservedRunningTime="2026-07-02 14:16:00.386186138 +0000 UTC m=+22.641591475"
Jul 02 14:16:01.293096 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.292829    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc51f60a-9cbb-4c55-9df2-faaa95d7822d" containerID="c4f40105796f944b5f10e58f465e42366a70ec8dd629ffdc8b55457b1b7dc553" exitCode=0
Jul 02 14:16:01.293745 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.292912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerDied","Data":"c4f40105796f944b5f10e58f465e42366a70ec8dd629ffdc8b55457b1b7dc553"}
Jul 02 14:16:01.295606 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.295589    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:16:01.295955 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.295935    2583 generic.go:356] "Generic (PLEG): container finished" podID="75a16613-faa9-4c72-ad72-e46f5e821947" containerID="390d4d8b3ed19d92ab09a7a9da186b8e76364ec7ced2529504bbaf2d99647a35" exitCode=1
Jul 02 14:16:01.296020 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.295981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"cce1e77bec5fa7ca8cdab14bd742db9601ec514b5d12ba861c07a105d50b911a"}
Jul 02 14:16:01.296065 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.296038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"6261cfa9ab7f0141879669878e06ec7f1eea90025bf8b1e887108278d7c26f29"}
Jul 02 14:16:01.296103 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.296068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"19b98be55f597713a9403c70a669c6cea1c43a083026439efd0a303380caf50b"}
Jul 02 14:16:01.296103 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.296096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerDied","Data":"390d4d8b3ed19d92ab09a7a9da186b8e76364ec7ced2529504bbaf2d99647a35"}
Jul 02 14:16:01.296172 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.296124    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"6a0e07699038601532ce0904b05eab64efa5f9f09909ae855e19696f8b487c5f"}
Jul 02 14:16:01.400321 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:01.400287    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 14:16:02.143923 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.143866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:02.144204 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.143866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:02.144204 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:02.143995    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:02.144204 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.144013    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:02.144204 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:02.144116    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:02.144204 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:02.144195    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:02.148393 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.148289    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T14:16:01.400308798Z","UUID":"4034ffc4-e20a-4ad4-8464-7b591a8893fb","Handler":null,"Name":"","Endpoint":""}
Jul 02 14:16:02.149892 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.149875    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 14:16:02.150008 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.149916    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 14:16:02.291133 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.291105    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:16:02.291828 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.291804    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:16:02.300205 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.300175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465" event={"ID":"cc2a1dba-58ab-488f-a113-e51af9710db3","Type":"ContainerStarted","Data":"1ea309ea2bf6b098fa5170e034752e915648821cfb78821b47764a8fd9e98e88"}
Jul 02 14:16:02.303253 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.303234    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:16:02.303666 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.303642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"1c2f34d090bdb588ed88f62963997461e4907ea3ba55dffd2f10a1411568be66"}
Jul 02 14:16:02.307632 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.307607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dmwts" event={"ID":"c88b0850-2950-401e-8569-fe7232e027a0","Type":"ContainerStarted","Data":"833a6c67281cd7b9f1af2e1902964414935704f34670d0f4e99c48dc9e44b09e"}
Jul 02 14:16:02.307858 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.307828    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:16:02.308282 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.308246    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dw2fn"
Jul 02 14:16:02.325429 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:02.325383    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dmwts" podStartSLOduration=5.762635584 podStartE2EDuration="24.325368094s" totalImagesPullingTime="18.56273251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:02.324624962 +0000 UTC m=+24.580030302" watchObservedRunningTime="2026-07-02 14:16:02.325368094 +0000 UTC m=+24.580773450"
Jul 02 14:16:04.143557 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:04.143349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:04.144025 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:04.143410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:04.144025 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:04.143662    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:04.144025 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:04.143687    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:04.144025 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:04.143349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:04.144025 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:04.143756    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:04.317872 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:04.317839    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:16:04.318322 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:04.318289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"2468d9339b382f94df1d74309cec3ab06b6eea6571daf665c637a833ccb1a83d"}
Jul 02 14:16:04.320520 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:04.320495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465" event={"ID":"cc2a1dba-58ab-488f-a113-e51af9710db3","Type":"ContainerStarted","Data":"94cad555aa43061a3a5c34a0c834884530490f74d28e96a3609b1518ead884c6"}
Jul 02 14:16:04.339137 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:04.339082    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5z465" podStartSLOduration=4.9444264239999995 podStartE2EDuration="26.339063762s" totalImagesPullingTime="21.394637338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:04.338824126 +0000 UTC m=+26.594229470" watchObservedRunningTime="2026-07-02 14:16:04.339063762 +0000 UTC m=+26.594469104"
Jul 02 14:16:06.143633 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.143467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:06.144285 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.143467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:06.144285 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:06.143707    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:06.144285 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:06.143796    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:06.144285 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.143467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:06.144285 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:06.143882    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:06.327981 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.327945    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:16:06.328357 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.328326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"81d0d7ffd05b67fc4b55a4e982e6f33e89dd5d9b52896c2b49584748cf53c862"}
Jul 02 14:16:06.329445 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.329418    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:16:06.332683 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.330307    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:16:06.332683 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.330787    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:16:06.332683 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.330830    2583 scope.go:117] "RemoveContainer" containerID="390d4d8b3ed19d92ab09a7a9da186b8e76364ec7ced2529504bbaf2d99647a35"
Jul 02 14:16:06.333367 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.333338    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc51f60a-9cbb-4c55-9df2-faaa95d7822d" containerID="42f11b3737d9fa1e2acf62421df248a6fe6aca95e56b5c604d7d6993705817ff" exitCode=0
Jul 02 14:16:06.333485 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.333378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerDied","Data":"42f11b3737d9fa1e2acf62421df248a6fe6aca95e56b5c604d7d6993705817ff"}
Jul 02 14:16:06.346691 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.346671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:16:06.349068 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:06.349008    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:16:07.340283 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.340255    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:16:07.340731 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.340660    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" event={"ID":"75a16613-faa9-4c72-ad72-e46f5e821947","Type":"ContainerStarted","Data":"7081ea298c3ae46cee6351d1d1147cf532f42ee68d3a3c4fb9805dc93ff98d07"}
Jul 02 14:16:07.374678 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.374627    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk" podStartSLOduration=9.959836455 podStartE2EDuration="29.374607642s" totalImagesPullingTime="19.414771187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:07.37419465 +0000 UTC m=+29.629599995" watchObservedRunningTime="2026-07-02 14:16:07.374607642 +0000 UTC m=+29.630012983"
Jul 02 14:16:07.596813 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.596717    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s62pd"]
Jul 02 14:16:07.597071 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.597048    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:07.597205 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:07.597183    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:07.600226 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.600197    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88xc5"]
Jul 02 14:16:07.600354 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.600308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:07.600431 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:07.600408    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:07.600804 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.600787    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4vgmc"]
Jul 02 14:16:07.600906 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:07.600896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:07.600991 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:07.600967    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:09.143916 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:09.143738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:09.144358 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:09.143739    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:09.144358 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:09.143994    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:09.144358 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:09.144093    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:09.144358 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:09.143739    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:09.144358 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:09.144208    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:09.347957 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:09.347923    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc51f60a-9cbb-4c55-9df2-faaa95d7822d" containerID="5480ffdbfd6e5ec2c6cb64f1797db1f18f3b9e326021a149c826aea7ea8ac692" exitCode=0
Jul 02 14:16:09.348127 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:09.347988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerDied","Data":"5480ffdbfd6e5ec2c6cb64f1797db1f18f3b9e326021a149c826aea7ea8ac692"}
Jul 02 14:16:11.144003 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:11.143923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:11.144003 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:11.143970    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:11.144504 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:11.143932    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:11.144504 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:11.144044    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:11.144504 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:11.144094    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:11.144504 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:11.144209    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:11.354043 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:11.354013    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc51f60a-9cbb-4c55-9df2-faaa95d7822d" containerID="bcb169ac828dbdcc9ef10765c1eb2b396b822607fe0722590c4f5f0ddee27d22" exitCode=0
Jul 02 14:16:11.354199 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:11.354070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerDied","Data":"bcb169ac828dbdcc9ef10765c1eb2b396b822607fe0722590c4f5f0ddee27d22"}
Jul 02 14:16:11.835631 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:11.835600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:11.835841 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:11.835784    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:11.835903 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:11.835860    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret podName:2aef6263-bcad-4789-89d3-c63178657d4f nodeName:}" failed. No retries permitted until 2026-07-02 14:16:43.835839979 +0000 UTC m=+66.091245302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret") pod "global-pull-secret-syncer-88xc5" (UID: "2aef6263-bcad-4789-89d3-c63178657d4f") : object "kube-system"/"original-pull-secret" not registered
Jul 02 14:16:11.936700 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:11.936660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:11.936855 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:11.936820    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:11.936905 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:11.936895    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs podName:46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:43.936877116 +0000 UTC m=+66.192282440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs") pod "network-metrics-daemon-4vgmc" (UID: "46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 14:16:12.037614 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:12.037577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:12.037818 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:12.037698    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 14:16:12.037818 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:12.037712    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 14:16:12.037818 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:12.037721    2583 projected.go:196] Error preparing data for projected volume kube-api-access-wxwbl for pod openshift-network-diagnostics/network-check-target-s62pd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:12.037818 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:12.037789    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl podName:feefb507-74b8-4f61-b823-359ea3ba6462 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:44.037757323 +0000 UTC m=+66.293162642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wxwbl" (UniqueName: "kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl") pod "network-check-target-s62pd" (UID: "feefb507-74b8-4f61-b823-359ea3ba6462") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 14:16:13.143211 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:13.143183    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:13.143865 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:13.143283    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:13.143865 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:13.143295    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s62pd" podUID="feefb507-74b8-4f61-b823-359ea3ba6462"
Jul 02 14:16:13.143865 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:13.143309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:13.143865 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:13.143386    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88xc5" podUID="2aef6263-bcad-4789-89d3-c63178657d4f"
Jul 02 14:16:13.143865 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:13.143529    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vgmc" podUID="46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18"
Jul 02 14:16:14.146404 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.146373    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-82.ec2.internal" event="NodeReady"
Jul 02 14:16:14.146861 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.146519    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 14:16:14.194396 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.194369    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gxzwx"]
Jul 02 14:16:14.197504 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.197486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.197604 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.197530    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rw985"]
Jul 02 14:16:14.200070 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.200050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 14:16:14.200166 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.200107    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v6rxw\""
Jul 02 14:16:14.200892 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.200878    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 14:16:14.201009 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.200993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.203491 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.203473    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 14:16:14.203571 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.203502    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 14:16:14.203740 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.203726    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pc47c\""
Jul 02 14:16:14.203794 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.203735    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 14:16:14.209095 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.209074    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gxzwx"]
Jul 02 14:16:14.210396 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.210365    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rw985"]
Jul 02 14:16:14.355538 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.355500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-config-volume\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.355720 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.355561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87f5b14c-698e-466b-a140-08b301139651-cert\") pod \"ingress-canary-rw985\" (UID: \"87f5b14c-698e-466b-a140-08b301139651\") " pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.355720 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.355590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.355720 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.355615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-tmp-dir\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.355877 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.355756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfvf\" (UniqueName: \"kubernetes.io/projected/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-kube-api-access-7gfvf\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.355877 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.355826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl4nv\" (UniqueName: \"kubernetes.io/projected/87f5b14c-698e-466b-a140-08b301139651-kube-api-access-vl4nv\") pod \"ingress-canary-rw985\" (UID: \"87f5b14c-698e-466b-a140-08b301139651\") " pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.456481 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.456450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.456639 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.456488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-tmp-dir\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.456639 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.456562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfvf\" (UniqueName: \"kubernetes.io/projected/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-kube-api-access-7gfvf\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.456639 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.456592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl4nv\" (UniqueName: \"kubernetes.io/projected/87f5b14c-698e-466b-a140-08b301139651-kube-api-access-vl4nv\") pod \"ingress-canary-rw985\" (UID: \"87f5b14c-698e-466b-a140-08b301139651\") " pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.456639 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:14.456627    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:16:14.456853 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:14.456694    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls podName:e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a nodeName:}" failed. No retries permitted until 2026-07-02 14:16:14.956673011 +0000 UTC m=+37.212078344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls") pod "dns-default-gxzwx" (UID: "e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a") : secret "dns-default-metrics-tls" not found
Jul 02 14:16:14.456853 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.456637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-config-volume\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.456853 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.456799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87f5b14c-698e-466b-a140-08b301139651-cert\") pod \"ingress-canary-rw985\" (UID: \"87f5b14c-698e-466b-a140-08b301139651\") " pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.457009 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.456943    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-tmp-dir\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.457553 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.457528    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-config-volume\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.461212 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.461193    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87f5b14c-698e-466b-a140-08b301139651-cert\") pod \"ingress-canary-rw985\" (UID: \"87f5b14c-698e-466b-a140-08b301139651\") " pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.466682 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.466629    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfvf\" (UniqueName: \"kubernetes.io/projected/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-kube-api-access-7gfvf\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.466808 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.466685    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl4nv\" (UniqueName: \"kubernetes.io/projected/87f5b14c-698e-466b-a140-08b301139651-kube-api-access-vl4nv\") pod \"ingress-canary-rw985\" (UID: \"87f5b14c-698e-466b-a140-08b301139651\") " pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.518730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.518694    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rw985"
Jul 02 14:16:14.961394 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:14.961355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:14.961574 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:14.961472    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:16:14.961574 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:14.961550    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls podName:e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a nodeName:}" failed. No retries permitted until 2026-07-02 14:16:15.961526863 +0000 UTC m=+38.216932182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls") pod "dns-default-gxzwx" (UID: "e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a") : secret "dns-default-metrics-tls" not found
Jul 02 14:16:15.143855 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.143815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:15.144023 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.143815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:15.144023 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.143819    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:15.146966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.146942    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 14:16:15.147418 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.146950    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 14:16:15.148506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.148327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s6j8d\""
Jul 02 14:16:15.148506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.148341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 14:16:15.148506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.148347    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 14:16:15.148506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.148406    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m6dwx\""
Jul 02 14:16:15.870751 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.870539    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rw985"]
Jul 02 14:16:15.969701 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:15.969670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:15.969899 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:15.969806    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:16:15.969899 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:15.969865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls podName:e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a nodeName:}" failed. No retries permitted until 2026-07-02 14:16:17.969852594 +0000 UTC m=+40.225257914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls") pod "dns-default-gxzwx" (UID: "e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a") : secret "dns-default-metrics-tls" not found
Jul 02 14:16:16.368705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:16.368626    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc51f60a-9cbb-4c55-9df2-faaa95d7822d" containerID="8f90ad6f3ec4e821d183b470b3f226d59102abf82895915c7e80f1db8a95eff7" exitCode=0
Jul 02 14:16:16.369158 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:16.368710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerDied","Data":"8f90ad6f3ec4e821d183b470b3f226d59102abf82895915c7e80f1db8a95eff7"}
Jul 02 14:16:16.369914 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:16.369883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rw985" event={"ID":"87f5b14c-698e-466b-a140-08b301139651","Type":"ContainerStarted","Data":"5890d0ef992fb0165cbaf876ffb6498875f66c60ad129578087d631887240fbb"}
Jul 02 14:16:17.375577 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:17.375541    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc51f60a-9cbb-4c55-9df2-faaa95d7822d" containerID="8005c68cb484c190ec380258ad1fe660141a4908b28fba53437cd9a0192b11b5" exitCode=0
Jul 02 14:16:17.376056 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:17.375606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerDied","Data":"8005c68cb484c190ec380258ad1fe660141a4908b28fba53437cd9a0192b11b5"}
Jul 02 14:16:17.987590 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:17.987357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:17.987851 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:17.987520    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:16:17.987851 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:17.987726    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls podName:e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a nodeName:}" failed. No retries permitted until 2026-07-02 14:16:21.987708545 +0000 UTC m=+44.243113874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls") pod "dns-default-gxzwx" (UID: "e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a") : secret "dns-default-metrics-tls" not found
Jul 02 14:16:18.243324 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.243238    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"]
Jul 02 14:16:18.262188 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.262162    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"]
Jul 02 14:16:18.262333 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.262276    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"
Jul 02 14:16:18.265153 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.265133    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 02 14:16:18.265264 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.265196    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 02 14:16:18.266439 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.266424    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ntdnd\""
Jul 02 14:16:18.380641 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.380607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" event={"ID":"bc51f60a-9cbb-4c55-9df2-faaa95d7822d","Type":"ContainerStarted","Data":"6b1764c36cd72dc9973db52573934b4ed0baf0305ae2050eba20afc0b5583228"}
Jul 02 14:16:18.389726 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.389703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvk94\" (UniqueName: \"kubernetes.io/projected/a3ef0e84-bffb-4237-b25e-f2ed433ce809-kube-api-access-qvk94\") pod \"migrator-786f9fbc5-hltm4\" (UID: \"a3ef0e84-bffb-4237-b25e-f2ed433ce809\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"
Jul 02 14:16:18.406755 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.406706    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gg5lm" podStartSLOduration=9.672206526 podStartE2EDuration="40.406690879s" totalImagesPullingTime="30.734484353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:18.406430419 +0000 UTC m=+40.661835760" watchObservedRunningTime="2026-07-02 14:16:18.406690879 +0000 UTC m=+40.662096220"
Jul 02 14:16:18.490429 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.490380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvk94\" (UniqueName: \"kubernetes.io/projected/a3ef0e84-bffb-4237-b25e-f2ed433ce809-kube-api-access-qvk94\") pod \"migrator-786f9fbc5-hltm4\" (UID: \"a3ef0e84-bffb-4237-b25e-f2ed433ce809\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"
Jul 02 14:16:18.511804 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.511734    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvk94\" (UniqueName: \"kubernetes.io/projected/a3ef0e84-bffb-4237-b25e-f2ed433ce809-kube-api-access-qvk94\") pod \"migrator-786f9fbc5-hltm4\" (UID: \"a3ef0e84-bffb-4237-b25e-f2ed433ce809\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"
Jul 02 14:16:18.570735 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.570700    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"
Jul 02 14:16:18.737679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.737643    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4"]
Jul 02 14:16:18.741847 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:18.741818    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ef0e84_bffb_4237_b25e_f2ed433ce809.slice/crio-9351437f5687a48aa681b5f5fd641a57433f8c6ea3a33fdc8a0bb6fe96104c75 WatchSource:0}: Error finding container 9351437f5687a48aa681b5f5fd641a57433f8c6ea3a33fdc8a0bb6fe96104c75: Status 404 returned error can't find the container with id 9351437f5687a48aa681b5f5fd641a57433f8c6ea3a33fdc8a0bb6fe96104c75
Jul 02 14:16:18.945621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.945586    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-twj2v"]
Jul 02 14:16:18.965954 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.965921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:18.969680 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.969645    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 14:16:18.969845 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.969700    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 14:16:18.969845 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.969737    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 14:16:18.970730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.970709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-twj2v"]
Jul 02 14:16:18.973475 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.973456    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 14:16:18.973675 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:18.973652    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wgq76\""
Jul 02 14:16:19.096304 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.096215    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.096304 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.096276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.096521 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.096313    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c165837-8308-4e76-92e8-a226d81bb246-crio-socket\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.096521 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.096329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c165837-8308-4e76-92e8-a226d81bb246-data-volume\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.096521 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.096408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8f9n\" (UniqueName: \"kubernetes.io/projected/8c165837-8308-4e76-92e8-a226d81bb246-kube-api-access-w8f9n\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.197680 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.197638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8f9n\" (UniqueName: \"kubernetes.io/projected/8c165837-8308-4e76-92e8-a226d81bb246-kube-api-access-w8f9n\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.197897 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.197694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.197897 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.197839    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:19.197897 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.197866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.197897 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.197888    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:19.69787283 +0000 UTC m=+41.953278152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:19.198092 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.197928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c165837-8308-4e76-92e8-a226d81bb246-crio-socket\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.198092 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.197931    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:16:19.198092 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.197947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c165837-8308-4e76-92e8-a226d81bb246-data-volume\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.198092 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.197997    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:19.697976598 +0000 UTC m=+41.953381933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : configmap "kube-rbac-proxy" not found
Jul 02 14:16:19.198208 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.198087    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8c165837-8308-4e76-92e8-a226d81bb246-crio-socket\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.198246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.198233    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8c165837-8308-4e76-92e8-a226d81bb246-data-volume\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.212659 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.212641    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8f9n\" (UniqueName: \"kubernetes.io/projected/8c165837-8308-4e76-92e8-a226d81bb246-kube-api-access-w8f9n\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.383985 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.383894    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rw985" event={"ID":"87f5b14c-698e-466b-a140-08b301139651","Type":"ContainerStarted","Data":"b7f57e9caff69342056b6ac2e7bdf4007bfd228551e6ce6557f3f619fe00374e"}
Jul 02 14:16:19.384943 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.384923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4" event={"ID":"a3ef0e84-bffb-4237-b25e-f2ed433ce809","Type":"ContainerStarted","Data":"9351437f5687a48aa681b5f5fd641a57433f8c6ea3a33fdc8a0bb6fe96104c75"}
Jul 02 14:16:19.403639 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.403580    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rw985" podStartSLOduration=2.679604512 podStartE2EDuration="5.403559862s" totalImagesPullingTime="2.72395535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:19.402829763 +0000 UTC m=+41.658235105" watchObservedRunningTime="2026-07-02 14:16:19.403559862 +0000 UTC m=+41.658965203"
Jul 02 14:16:19.702593 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.702509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.702809 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:19.702607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:19.702809 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.702635    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:16:19.702809 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.702707    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:20.70268814 +0000 UTC m=+42.958093473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : configmap "kube-rbac-proxy" not found
Jul 02 14:16:19.702809 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.702728    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:19.702809 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:19.702788    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:20.702752337 +0000 UTC m=+42.958157656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:20.710643 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:20.710607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:20.711015 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:20.710680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:20.711015 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:20.710777    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:20.711015 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:20.710794    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:16:20.711015 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:20.710844    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:22.710829769 +0000 UTC m=+44.966235087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:20.711015 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:20.710859    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:22.710853327 +0000 UTC m=+44.966258646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : configmap "kube-rbac-proxy" not found
Jul 02 14:16:21.394227 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.394190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4" event={"ID":"a3ef0e84-bffb-4237-b25e-f2ed433ce809","Type":"ContainerStarted","Data":"641536a1229ed6b8d373264fa203628adfa80f3ed00aa3edffbde209ef794833"}
Jul 02 14:16:21.394427 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.394262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4" event={"ID":"a3ef0e84-bffb-4237-b25e-f2ed433ce809","Type":"ContainerStarted","Data":"0a574ef6016caef4608641746f5cf4dfa69e8cbc2ab647b0409a94f32068f652"}
Jul 02 14:16:21.413541 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.413490    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-hltm4" podStartSLOduration=1.679647747 podStartE2EDuration="3.413474879s" totalImagesPullingTime="1.733827132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:21.411696256 +0000 UTC m=+43.667101596" watchObservedRunningTime="2026-07-02 14:16:21.413474879 +0000 UTC m=+43.668880219"
Jul 02 14:16:21.686658 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.686582    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"]
Jul 02 14:16:21.709326 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.709291    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"]
Jul 02 14:16:21.709477 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.709405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.713121 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.713093    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 14:16:21.713482 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.713123    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 14:16:21.713482 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.713124    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 14:16:21.713482 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.713137    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j4ss2\""
Jul 02 14:16:21.714431 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.714410    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 14:16:21.716891 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.716869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2af5d7e1-043a-426b-b1ef-b3e2d634905b-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.716993 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.716910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67h8\" (UniqueName: \"kubernetes.io/projected/2af5d7e1-043a-426b-b1ef-b3e2d634905b-kube-api-access-k67h8\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.716993 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.716972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2af5d7e1-043a-426b-b1ef-b3e2d634905b-signing-key\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.734623 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.734602    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5tl5_292e4555-5517-496c-b83a-b1e4090e0579/dns-node-resolver/0.log"
Jul 02 14:16:21.817547 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.817516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2af5d7e1-043a-426b-b1ef-b3e2d634905b-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.817703 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.817558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67h8\" (UniqueName: \"kubernetes.io/projected/2af5d7e1-043a-426b-b1ef-b3e2d634905b-kube-api-access-k67h8\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.817703 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.817592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2af5d7e1-043a-426b-b1ef-b3e2d634905b-signing-key\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.818239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.818214    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2af5d7e1-043a-426b-b1ef-b3e2d634905b-signing-cabundle\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.819936 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.819920    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2af5d7e1-043a-426b-b1ef-b3e2d634905b-signing-key\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:21.825926 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:21.825906    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67h8\" (UniqueName: \"kubernetes.io/projected/2af5d7e1-043a-426b-b1ef-b3e2d634905b-kube-api-access-k67h8\") pod \"service-ca-7bb4f4bcc7-5l5kj\" (UID: \"2af5d7e1-043a-426b-b1ef-b3e2d634905b\") " pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:22.018316 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:22.018282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"
Jul 02 14:16:22.018461 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:22.018393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:22.018564 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:22.018546    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:16:22.018614 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:22.018605    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls podName:e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a nodeName:}" failed. No retries permitted until 2026-07-02 14:16:30.018591357 +0000 UTC m=+52.273996677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls") pod "dns-default-gxzwx" (UID: "e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a") : secret "dns-default-metrics-tls" not found
Jul 02 14:16:22.174589 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:22.174558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj"]
Jul 02 14:16:22.177654 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:22.177624    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af5d7e1_043a_426b_b1ef_b3e2d634905b.slice/crio-2931b1e159b82fcb61e4b9ddf521c07678efc03e0cfaac1e2ecaf9045877ede1 WatchSource:0}: Error finding container 2931b1e159b82fcb61e4b9ddf521c07678efc03e0cfaac1e2ecaf9045877ede1: Status 404 returned error can't find the container with id 2931b1e159b82fcb61e4b9ddf521c07678efc03e0cfaac1e2ecaf9045877ede1
Jul 02 14:16:22.398206 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:22.398123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj" event={"ID":"2af5d7e1-043a-426b-b1ef-b3e2d634905b","Type":"ContainerStarted","Data":"2931b1e159b82fcb61e4b9ddf521c07678efc03e0cfaac1e2ecaf9045877ede1"}
Jul 02 14:16:22.724198 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:22.724165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:22.724539 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:22.724226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:22.724539 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:22.724304    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 14:16:22.724539 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:22.724315    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:22.724539 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:22.724358    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:26.724344665 +0000 UTC m=+48.979749983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : configmap "kube-rbac-proxy" not found
Jul 02 14:16:22.724539 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:22.724371    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:26.724364413 +0000 UTC m=+48.979769732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:22.741750 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:22.741730    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-llvbz_36221c70-cb82-48b5-8d8c-d6babeccf3b5/node-ca/0.log"
Jul 02 14:16:23.734474 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:23.734442    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rw985_87f5b14c-698e-466b-a140-08b301139651/serve-healthcheck-canary/0.log"
Jul 02 14:16:23.935013 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:23.934976    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-hltm4_a3ef0e84-bffb-4237-b25e-f2ed433ce809/migrator/0.log"
Jul 02 14:16:24.134667 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:24.134584    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-hltm4_a3ef0e84-bffb-4237-b25e-f2ed433ce809/graceful-termination/0.log"
Jul 02 14:16:25.407412 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:25.407372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj" event={"ID":"2af5d7e1-043a-426b-b1ef-b3e2d634905b","Type":"ContainerStarted","Data":"295d70275cd17b9d7407fd286c875c7ea87078c37a55b4a294c872b532cbd03a"}
Jul 02 14:16:25.437512 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:25.437462    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7bb4f4bcc7-5l5kj" podStartSLOduration=1.824787888 podStartE2EDuration="4.437444999s" totalImagesPullingTime="2.612657111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:25.43716917 +0000 UTC m=+47.692574592" watchObservedRunningTime="2026-07-02 14:16:25.437444999 +0000 UTC m=+47.692850340"
Jul 02 14:16:26.759257 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:26.759208    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:26.759626 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:26.759301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:26.759942 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:26.759920    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8c165837-8308-4e76-92e8-a226d81bb246-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:26.760053 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:26.760040    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:26.760134 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:26.760102    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls podName:8c165837-8308-4e76-92e8-a226d81bb246 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:34.760084303 +0000 UTC m=+57.015489630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls") pod "insights-runtime-extractor-twj2v" (UID: "8c165837-8308-4e76-92e8-a226d81bb246") : secret "insights-runtime-extractor-tls" not found
Jul 02 14:16:30.084168 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:30.084130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:30.084675 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:30.084283    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 14:16:30.084675 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:30.084347    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls podName:e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a nodeName:}" failed. No retries permitted until 2026-07-02 14:16:46.084329751 +0000 UTC m=+68.339735075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls") pod "dns-default-gxzwx" (UID: "e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a") : secret "dns-default-metrics-tls" not found
Jul 02 14:16:34.818463 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:34.818424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:34.820886 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:34.820861    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8c165837-8308-4e76-92e8-a226d81bb246-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-twj2v\" (UID: \"8c165837-8308-4e76-92e8-a226d81bb246\") " pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:34.875818 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:34.875783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-twj2v"
Jul 02 14:16:35.000016 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:34.999981    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-twj2v"]
Jul 02 14:16:35.002679 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:35.002647    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c165837_8308_4e76_92e8_a226d81bb246.slice/crio-d65b991c8b205e5baba772610d5bcfed0a871940fdb0ab6273638aacf2719d45 WatchSource:0}: Error finding container d65b991c8b205e5baba772610d5bcfed0a871940fdb0ab6273638aacf2719d45: Status 404 returned error can't find the container with id d65b991c8b205e5baba772610d5bcfed0a871940fdb0ab6273638aacf2719d45
Jul 02 14:16:35.433562 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:35.433516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twj2v" event={"ID":"8c165837-8308-4e76-92e8-a226d81bb246","Type":"ContainerStarted","Data":"d65b991c8b205e5baba772610d5bcfed0a871940fdb0ab6273638aacf2719d45"}
Jul 02 14:16:38.360069 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:38.360035    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2cfk"
Jul 02 14:16:40.449599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:40.449564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twj2v" event={"ID":"8c165837-8308-4e76-92e8-a226d81bb246","Type":"ContainerStarted","Data":"63b5f2ac8145177337551e195249ee2f99da445fa42b0718766142d3b656fd79"}
Jul 02 14:16:40.449961 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:40.449621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twj2v" event={"ID":"8c165837-8308-4e76-92e8-a226d81bb246","Type":"ContainerStarted","Data":"b434c4d13d9d21689fd0b53994e4b8e1445d8725df4bb20dc09bb8c7a72dc105"}
Jul 02 14:16:43.461496 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.461460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-twj2v" event={"ID":"8c165837-8308-4e76-92e8-a226d81bb246","Type":"ContainerStarted","Data":"152d8567f3ef4b4470873920fa3030287c4bd602c5e73f69ce110ad1a0370d01"}
Jul 02 14:16:43.482933 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.481048    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-twj2v" podStartSLOduration=17.704827054 podStartE2EDuration="25.4810301s" totalImagesPullingTime="7.776203046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:43.479906544 +0000 UTC m=+65.735311888" watchObservedRunningTime="2026-07-02 14:16:43.4810301 +0000 UTC m=+65.736435442"
Jul 02 14:16:43.891927 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.891839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:43.894438 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.894419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 14:16:43.904671 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.904646    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aef6263-bcad-4789-89d3-c63178657d4f-original-pull-secret\") pod \"global-pull-secret-syncer-88xc5\" (UID: \"2aef6263-bcad-4789-89d3-c63178657d4f\") " pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:43.964014 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.963980    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88xc5"
Jul 02 14:16:43.992511 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.992480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:43.995245 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:43.995229    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 14:16:44.005692 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.005664    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18-metrics-certs\") pod \"network-metrics-daemon-4vgmc\" (UID: \"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18\") " pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:44.087330 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.087298    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88xc5"]
Jul 02 14:16:44.090433 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:44.090400    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aef6263_bcad_4789_89d3_c63178657d4f.slice/crio-40eef38124c87f571a79f4e168c2e925291e8b2d80ba1bdf75f2c8d4169b1ed5 WatchSource:0}: Error finding container 40eef38124c87f571a79f4e168c2e925291e8b2d80ba1bdf75f2c8d4169b1ed5: Status 404 returned error can't find the container with id 40eef38124c87f571a79f4e168c2e925291e8b2d80ba1bdf75f2c8d4169b1ed5
Jul 02 14:16:44.092832 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.092812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:44.095457 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.095436    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 14:16:44.106679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.106652    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 14:16:44.116333 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.116304    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxwbl\" (UniqueName: \"kubernetes.io/projected/feefb507-74b8-4f61-b823-359ea3ba6462-kube-api-access-wxwbl\") pod \"network-check-target-s62pd\" (UID: \"feefb507-74b8-4f61-b823-359ea3ba6462\") " pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:44.154463 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.154381    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"]
Jul 02 14:16:44.158592 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.158558    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.163151 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.163127    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"]
Jul 02 14:16:44.164522 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.164501    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 14:16:44.164843 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.164821    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 14:16:44.164964 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.164926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 02 14:16:44.165480 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.165464    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qdbn6\""
Jul 02 14:16:44.166479 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.166462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.167933 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.167915    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 14:16:44.168740 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.168726    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 02 14:16:44.169000 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.168985    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 02 14:16:44.169267 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.169256    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 02 14:16:44.169613 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.169601    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 02 14:16:44.180805 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.180784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"]
Jul 02 14:16:44.193390 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.193367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b97d0a9e-c0cc-4fd6-b0f9-5564e0018959-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668cdf8556-bdmm5\" (UID: \"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.193487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.193426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs4jj\" (UniqueName: \"kubernetes.io/projected/b97d0a9e-c0cc-4fd6-b0f9-5564e0018959-kube-api-access-hs4jj\") pod \"managed-serviceaccount-addon-agent-668cdf8556-bdmm5\" (UID: \"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.197740 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.197716    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"]
Jul 02 14:16:44.259603 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.259571    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m6dwx\""
Jul 02 14:16:44.267817 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.267796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vgmc"
Jul 02 14:16:44.271896 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.271880    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s6j8d\""
Jul 02 14:16:44.280065 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.280044    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:44.294381 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.294605 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs4jj\" (UniqueName: \"kubernetes.io/projected/b97d0a9e-c0cc-4fd6-b0f9-5564e0018959-kube-api-access-hs4jj\") pod \"managed-serviceaccount-addon-agent-668cdf8556-bdmm5\" (UID: \"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.294708 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-ca\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.294708 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrhz\" (UniqueName: \"kubernetes.io/projected/1c2ea850-9e94-4b19-b2b7-efdf72612923-kube-api-access-hhrhz\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.294836 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b97d0a9e-c0cc-4fd6-b0f9-5564e0018959-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668cdf8556-bdmm5\" (UID: \"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.294836 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-hub\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.294939 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.294991 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.294951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ea850-9e94-4b19-b2b7-efdf72612923-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.297337 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.297316    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b97d0a9e-c0cc-4fd6-b0f9-5564e0018959-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-668cdf8556-bdmm5\" (UID: \"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.321528 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.321466    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs4jj\" (UniqueName: \"kubernetes.io/projected/b97d0a9e-c0cc-4fd6-b0f9-5564e0018959-kube-api-access-hs4jj\") pod \"managed-serviceaccount-addon-agent-668cdf8556-bdmm5\" (UID: \"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.395635 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.395606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ea850-9e94-4b19-b2b7-efdf72612923-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.395635 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.395637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.395852 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.395694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-ca\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.395852 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.395710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrhz\" (UniqueName: \"kubernetes.io/projected/1c2ea850-9e94-4b19-b2b7-efdf72612923-kube-api-access-hhrhz\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.395852 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.395732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-hub\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.395852 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.395751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.396480 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.396454    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ea850-9e94-4b19-b2b7-efdf72612923-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.398315 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.398292    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.398424 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.398315    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-ca\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.398662 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.398635    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.398704 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.398676    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1c2ea850-9e94-4b19-b2b7-efdf72612923-hub\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.414349 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.414330    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4vgmc"]
Jul 02 14:16:44.415886 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.415870    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrhz\" (UniqueName: \"kubernetes.io/projected/1c2ea850-9e94-4b19-b2b7-efdf72612923-kube-api-access-hhrhz\") pod \"cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq\" (UID: \"1c2ea850-9e94-4b19-b2b7-efdf72612923\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.416559 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:44.416541    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a6dee6_6ab6_4cd8_a2ff_468d19dfaf18.slice/crio-80f2b216a6c4c51a727f6ef66e459fbc528080f7fa23f29a56ba0b01e920db4e WatchSource:0}: Error finding container 80f2b216a6c4c51a727f6ef66e459fbc528080f7fa23f29a56ba0b01e920db4e: Status 404 returned error can't find the container with id 80f2b216a6c4c51a727f6ef66e459fbc528080f7fa23f29a56ba0b01e920db4e
Jul 02 14:16:44.430105 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.430085    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s62pd"]
Jul 02 14:16:44.433076 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:44.433054    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeefb507_74b8_4f61_b823_359ea3ba6462.slice/crio-eba843df45857d05c87ecd802dda109bb7a2477c90e45b99348b172d852e928e WatchSource:0}: Error finding container eba843df45857d05c87ecd802dda109bb7a2477c90e45b99348b172d852e928e: Status 404 returned error can't find the container with id eba843df45857d05c87ecd802dda109bb7a2477c90e45b99348b172d852e928e
Jul 02 14:16:44.465569 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.465542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88xc5" event={"ID":"2aef6263-bcad-4789-89d3-c63178657d4f","Type":"ContainerStarted","Data":"40eef38124c87f571a79f4e168c2e925291e8b2d80ba1bdf75f2c8d4169b1ed5"}
Jul 02 14:16:44.466678 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.466645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4vgmc" event={"ID":"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18","Type":"ContainerStarted","Data":"80f2b216a6c4c51a727f6ef66e459fbc528080f7fa23f29a56ba0b01e920db4e"}
Jul 02 14:16:44.467742 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.467716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s62pd" event={"ID":"feefb507-74b8-4f61-b823-359ea3ba6462","Type":"ContainerStarted","Data":"eba843df45857d05c87ecd802dda109bb7a2477c90e45b99348b172d852e928e"}
Jul 02 14:16:44.476186 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.476168    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"
Jul 02 14:16:44.482280 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.482261    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"
Jul 02 14:16:44.604361 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.604331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5"]
Jul 02 14:16:44.606923 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:44.606902    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb97d0a9e_c0cc_4fd6_b0f9_5564e0018959.slice/crio-8804615e6baa11e58a65ca40448c62d821d6ffff62aaf14322113ace0b5f78c2 WatchSource:0}: Error finding container 8804615e6baa11e58a65ca40448c62d821d6ffff62aaf14322113ace0b5f78c2: Status 404 returned error can't find the container with id 8804615e6baa11e58a65ca40448c62d821d6ffff62aaf14322113ace0b5f78c2
Jul 02 14:16:44.630155 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:44.630132    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq"]
Jul 02 14:16:44.632902 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:44.632878    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2ea850_9e94_4b19_b2b7_efdf72612923.slice/crio-6ae7ea87209b05ed08d5820816089e0344e14b35b27fa7369513adef52f488ea WatchSource:0}: Error finding container 6ae7ea87209b05ed08d5820816089e0344e14b35b27fa7369513adef52f488ea: Status 404 returned error can't find the container with id 6ae7ea87209b05ed08d5820816089e0344e14b35b27fa7369513adef52f488ea
Jul 02 14:16:45.473236 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:45.473193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5" event={"ID":"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959","Type":"ContainerStarted","Data":"8804615e6baa11e58a65ca40448c62d821d6ffff62aaf14322113ace0b5f78c2"}
Jul 02 14:16:45.474607 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:45.474579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq" event={"ID":"1c2ea850-9e94-4b19-b2b7-efdf72612923","Type":"ContainerStarted","Data":"6ae7ea87209b05ed08d5820816089e0344e14b35b27fa7369513adef52f488ea"}
Jul 02 14:16:46.111038 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:46.110998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:46.113632 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:46.113607    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a-metrics-tls\") pod \"dns-default-gxzwx\" (UID: \"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a\") " pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:46.310629 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:46.310604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-v6rxw\""
Jul 02 14:16:46.318034 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:46.318009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:46.656181 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:46.651063    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gxzwx"]
Jul 02 14:16:47.489064 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:47.488975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxzwx" event={"ID":"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a","Type":"ContainerStarted","Data":"10f0cca2bff71ad163f3af9733e2d2b6f24b0f554a956fbf595a7c489459ae4f"}
Jul 02 14:16:47.490912 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:47.490876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4vgmc" event={"ID":"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18","Type":"ContainerStarted","Data":"f1720a0ac43bb54a7bcb3c56c921918939512c6af7d98e7cb8e815b52c06ab76"}
Jul 02 14:16:52.515808 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:52.515699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88xc5" event={"ID":"2aef6263-bcad-4789-89d3-c63178657d4f","Type":"ContainerStarted","Data":"6c31572f74dcf6805e375a99fd950b425b33c7b5281014c39b194b12b6a63fd6"}
Jul 02 14:16:52.517833 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:52.517808    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4vgmc" event={"ID":"46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18","Type":"ContainerStarted","Data":"e9dee856c975ab8e17fae943a93703c2a534182b420feb4cb7ab977c7aacddd2"}
Jul 02 14:16:52.536323 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:52.536272    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-88xc5" podStartSLOduration=66.393119213 podStartE2EDuration="1m14.536259886s" totalImagesPullingTime="8.143140673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:52.53480463 +0000 UTC m=+74.790209972" watchObservedRunningTime="2026-07-02 14:16:52.536259886 +0000 UTC m=+74.791665228"
Jul 02 14:16:52.552953 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:52.552903    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4vgmc" podStartSLOduration=72.51504008 podStartE2EDuration="1m14.552879092s" totalImagesPullingTime="2.037839012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:52.552345434 +0000 UTC m=+74.807750776" watchObservedRunningTime="2026-07-02 14:16:52.552879092 +0000 UTC m=+74.808284432"
Jul 02 14:16:53.522797 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.522744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s62pd" event={"ID":"feefb507-74b8-4f61-b823-359ea3ba6462","Type":"ContainerStarted","Data":"77b7af6e0355463b5d0050529c80accc6e12d8e97f283360ec83c6089fd947ee"}
Jul 02 14:16:53.523227 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.522856    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:16:53.524259 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.524231    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5" event={"ID":"b97d0a9e-c0cc-4fd6-b0f9-5564e0018959","Type":"ContainerStarted","Data":"2039cab4ab96726c510ab697a8ad14147e9ac8b48b27d5e7345f245e0af6a563"}
Jul 02 14:16:53.526087 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.526068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq" event={"ID":"1c2ea850-9e94-4b19-b2b7-efdf72612923","Type":"ContainerStarted","Data":"accb88ff32eabc1861d1179312abb4b77e7a9243576527fad90085e88888340e"}
Jul 02 14:16:53.526163 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.526095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq" event={"ID":"1c2ea850-9e94-4b19-b2b7-efdf72612923","Type":"ContainerStarted","Data":"99e7fae587cf6582b0eec459a56027c8810034ceffed5a08719a24e90747349e"}
Jul 02 14:16:53.526163 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.526108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq" event={"ID":"1c2ea850-9e94-4b19-b2b7-efdf72612923","Type":"ContainerStarted","Data":"afc1b8f6ccad5f5bc5c2b108a328777535fb5ca02d3317e2f1ad1ddc9a34ecd6"}
Jul 02 14:16:53.527817 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.527794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxzwx" event={"ID":"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a","Type":"ContainerStarted","Data":"1c0cc1ce45fcf92da41eef75466d35eb5a7f61b771bd8f89e9fe8f1d3c430f9e"}
Jul 02 14:16:53.527905 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.527833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gxzwx" event={"ID":"e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a","Type":"ContainerStarted","Data":"6c955e80153a3bfbe4a8b1edf37e166bd027c9357ecfecef01c078d10effb0e8"}
Jul 02 14:16:53.543228 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.543187    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s62pd" podStartSLOduration=67.733978942 podStartE2EDuration="1m15.543173274s" totalImagesPullingTime="7.809194332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:53.541492638 +0000 UTC m=+75.796897980" watchObservedRunningTime="2026-07-02 14:16:53.543173274 +0000 UTC m=+75.798578614"
Jul 02 14:16:53.559660 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.559618    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gxzwx" podStartSLOduration=33.276586465 podStartE2EDuration="39.559605625s" totalImagesPullingTime="6.28301916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:53.559036293 +0000 UTC m=+75.814441633" watchObservedRunningTime="2026-07-02 14:16:53.559605625 +0000 UTC m=+75.815010966"
Jul 02 14:16:53.578379 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.578347    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-668cdf8556-bdmm5" podStartSLOduration=1.342630775 podStartE2EDuration="9.578334337s" totalImagesPullingTime="8.235703562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:53.578179601 +0000 UTC m=+75.833584942" watchObservedRunningTime="2026-07-02 14:16:53.578334337 +0000 UTC m=+75.833739680"
Jul 02 14:16:53.600860 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:53.600821    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f5b65cdd9-jkdjq" podStartSLOduration=1.94198423 podStartE2EDuration="9.60081092s" totalImagesPullingTime="7.65882669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:16:53.600185195 +0000 UTC m=+75.855590536" watchObservedRunningTime="2026-07-02 14:16:53.60081092 +0000 UTC m=+75.856216261"
Jul 02 14:16:54.531679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:54.531649    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:16:56.993730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:56.993699    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j4p6q"]
Jul 02 14:16:56.996614 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:56.996593    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.005032 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:57.004731    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-132-82.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-132-82.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret"
Jul 02 14:16:57.007431 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.007399    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 14:16:57.007611 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.007593    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 14:16:57.007687 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.007669    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 14:16:57.008731 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.008715    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xv2pg\""
Jul 02 14:16:57.008810 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.008720    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 14:16:57.012841 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.012824    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 14:16:57.100985 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.100951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl9pv\" (UniqueName: \"kubernetes.io/projected/75597c68-a05f-409b-b0e9-bea48443afe2-kube-api-access-zl9pv\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101130 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101130 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101130 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-tls\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101255 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-root\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101255 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-textfile\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101255 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75597c68-a05f-409b-b0e9-bea48443afe2-metrics-client-ca\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101255 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-sys\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.101376 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.101258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-wtmp\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.201783 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.201783 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-tls\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-root\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-textfile\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75597c68-a05f-409b-b0e9-bea48443afe2-metrics-client-ca\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-sys\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-wtmp\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201888    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-root\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202006 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.201948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl9pv\" (UniqueName: \"kubernetes.io/projected/75597c68-a05f-409b-b0e9-bea48443afe2-kube-api-access-zl9pv\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202386 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.202005    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-sys\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202386 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.202067    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-wtmp\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202386 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.202248    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-textfile\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202541 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.202397    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-accelerators-collector-config\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.202599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.202552    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75597c68-a05f-409b-b0e9-bea48443afe2-metrics-client-ca\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.204340 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.204322    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-tls\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:57.216610 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:57.216584    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl9pv\" (UniqueName: \"kubernetes.io/projected/75597c68-a05f-409b-b0e9-bea48443afe2-kube-api-access-zl9pv\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:58.202329 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:58.202227    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition
Jul 02 14:16:58.202329 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:16:58.202295    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-kube-rbac-proxy-config podName:75597c68-a05f-409b-b0e9-bea48443afe2 nodeName:}" failed. No retries permitted until 2026-07-02 14:16:58.702279633 +0000 UTC m=+80.957684952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-kube-rbac-proxy-config") pod "node-exporter-j4p6q" (UID: "75597c68-a05f-409b-b0e9-bea48443afe2") : failed to sync secret cache: timed out waiting for the condition
Jul 02 14:16:58.367571 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:58.367545    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 14:16:58.715928 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:58.715887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:58.718111 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:58.718093    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/75597c68-a05f-409b-b0e9-bea48443afe2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j4p6q\" (UID: \"75597c68-a05f-409b-b0e9-bea48443afe2\") " pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:58.808314 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:58.808282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j4p6q"
Jul 02 14:16:58.822084 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:16:58.822061    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75597c68_a05f_409b_b0e9_bea48443afe2.slice/crio-8a4c4027f3667f1d2350e88e759b88d5039a6248819b7ca805840265ec028764 WatchSource:0}: Error finding container 8a4c4027f3667f1d2350e88e759b88d5039a6248819b7ca805840265ec028764: Status 404 returned error can't find the container with id 8a4c4027f3667f1d2350e88e759b88d5039a6248819b7ca805840265ec028764
Jul 02 14:16:59.555369 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:16:59.555334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4p6q" event={"ID":"75597c68-a05f-409b-b0e9-bea48443afe2","Type":"ContainerStarted","Data":"8a4c4027f3667f1d2350e88e759b88d5039a6248819b7ca805840265ec028764"}
Jul 02 14:17:01.563475 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:17:01.563442    2583 generic.go:356] "Generic (PLEG): container finished" podID="75597c68-a05f-409b-b0e9-bea48443afe2" containerID="4cde473f02ba1f6dc13ca871ce22ce8e78731e12845c980c15091c82cda0e200" exitCode=0
Jul 02 14:17:01.563891 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:17:01.563530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4p6q" event={"ID":"75597c68-a05f-409b-b0e9-bea48443afe2","Type":"ContainerDied","Data":"4cde473f02ba1f6dc13ca871ce22ce8e78731e12845c980c15091c82cda0e200"}
Jul 02 14:17:02.569088 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:17:02.569054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4p6q" event={"ID":"75597c68-a05f-409b-b0e9-bea48443afe2","Type":"ContainerStarted","Data":"8ea30829c215f19f6d346c42e8c2e4b47ca61e9a0aee69bb754aebbd800abdf2"}
Jul 02 14:17:02.569460 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:17:02.569103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j4p6q" event={"ID":"75597c68-a05f-409b-b0e9-bea48443afe2","Type":"ContainerStarted","Data":"2646dcd79adad92d0167f7a2d5e2e0c641d9905e6d6a88a0a68e8712fd3e71cd"}
Jul 02 14:17:02.592239 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:17:02.592188    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j4p6q" podStartSLOduration=4.826488759 podStartE2EDuration="6.592174028s" totalImagesPullingTime="1.703416334s" totalInitContainerRuntime="62.268935ms" isStatefulPod=true podCreationTimestamp="2026-07-02 14:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:17:02.590210766 +0000 UTC m=+84.845616107" watchObservedRunningTime="2026-07-02 14:17:02.592174028 +0000 UTC m=+84.847579368"
Jul 02 14:17:04.538033 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:17:04.538000    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gxzwx"
Jul 02 14:17:24.534528 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:17:24.534497    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s62pd"
Jul 02 14:20:38.083270 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:38.083240    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:20:38.087999 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:38.087964    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:20:38.089644 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:38.089621    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 14:20:55.000301 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.000264    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-d6859"]
Jul 02 14:20:55.003493 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.003477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.008229 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.008082    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 02 14:20:55.008229 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.008090    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 02 14:20:55.008229 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.008124    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 02 14:20:55.008229 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.008211    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zcw29\""
Jul 02 14:20:55.008529 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.008213    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 02 14:20:55.008529 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.008370    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 02 14:20:55.021083 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.021014    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-d6859"]
Jul 02 14:20:55.115686 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.115658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-certificates\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.115686 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.115692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ac3fea19-ca33-4254-b0c7-630ce58ec654-cabundle0\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.115936 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.115728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vw7t\" (UniqueName: \"kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-kube-api-access-7vw7t\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.216922 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.216890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vw7t\" (UniqueName: \"kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-kube-api-access-7vw7t\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.217120 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.216955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-certificates\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.217120 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.216978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ac3fea19-ca33-4254-b0c7-630ce58ec654-cabundle0\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.217120 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:20:55.217108    2583 secret.go:281] references non-existent secret key: ca.crt
Jul 02 14:20:55.217274 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:20:55.217124    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 02 14:20:55.217274 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:20:55.217134    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-d6859: references non-existent secret key: ca.crt
Jul 02 14:20:55.217274 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:20:55.217196    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-certificates podName:ac3fea19-ca33-4254-b0c7-630ce58ec654 nodeName:}" failed. No retries permitted until 2026-07-02 14:20:55.717174986 +0000 UTC m=+317.972580316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-certificates") pod "keda-operator-68bff59c-d6859" (UID: "ac3fea19-ca33-4254-b0c7-630ce58ec654") : references non-existent secret key: ca.crt
Jul 02 14:20:55.217644 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.217627    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ac3fea19-ca33-4254-b0c7-630ce58ec654-cabundle0\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.236695 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.236664    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vw7t\" (UniqueName: \"kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-kube-api-access-7vw7t\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.721125 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.721093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-certificates\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.723533 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.723503    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ac3fea19-ca33-4254-b0c7-630ce58ec654-certificates\") pod \"keda-operator-68bff59c-d6859\" (UID: \"ac3fea19-ca33-4254-b0c7-630ce58ec654\") " pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:55.912993 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:55.912958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:20:56.042912 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:56.042884    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-d6859"]
Jul 02 14:20:56.045979 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:20:56.045953    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3fea19_ca33_4254_b0c7_630ce58ec654.slice/crio-b81bd9f6dbcc7f7535912b3dbebe2781cba2cee88b4906401735d38e25c3443f WatchSource:0}: Error finding container b81bd9f6dbcc7f7535912b3dbebe2781cba2cee88b4906401735d38e25c3443f: Status 404 returned error can't find the container with id b81bd9f6dbcc7f7535912b3dbebe2781cba2cee88b4906401735d38e25c3443f
Jul 02 14:20:56.047235 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:56.047219    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:20:56.306581 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:20:56.306504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-d6859" event={"ID":"ac3fea19-ca33-4254-b0c7-630ce58ec654","Type":"ContainerStarted","Data":"b81bd9f6dbcc7f7535912b3dbebe2781cba2cee88b4906401735d38e25c3443f"}
Jul 02 14:21:03.331415 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:21:03.331381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-d6859" event={"ID":"ac3fea19-ca33-4254-b0c7-630ce58ec654","Type":"ContainerStarted","Data":"d14e7aa0174d2955e87fbae8dd86f6cc4987ab3be6934ca3d557bfd676dfe279"}
Jul 02 14:21:03.331840 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:21:03.331493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:21:03.350523 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:21:03.350477    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-d6859" podStartSLOduration=3.015003181 podStartE2EDuration="9.350460279s" totalImagesPullingTime="6.335457098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:21:03.349486565 +0000 UTC m=+325.604891905" watchObservedRunningTime="2026-07-02 14:21:03.350460279 +0000 UTC m=+325.605865619"
Jul 02 14:21:24.337135 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:21:24.337102    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-d6859"
Jul 02 14:22:03.443332 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.443303    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-gphs5"]
Jul 02 14:22:03.446287 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.446271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.453125 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.453101    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 02 14:22:03.454570 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.454548    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 14:22:03.454698 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.454548    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 14:22:03.454698 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.454553    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-cxtlq\""
Jul 02 14:22:03.466830 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.466804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-gphs5"]
Jul 02 14:22:03.475481 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.475458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/466d2c69-6376-4e86-beed-c5a07c49f6da-cert\") pod \"kserve-controller-manager-7b9c77fff9-gphs5\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") " pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.475614 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.475504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkv6\" (UniqueName: \"kubernetes.io/projected/466d2c69-6376-4e86-beed-c5a07c49f6da-kube-api-access-vtkv6\") pod \"kserve-controller-manager-7b9c77fff9-gphs5\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") " pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.476945 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.476926    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2wcb7"]
Jul 02 14:22:03.485537 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.485511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.488872 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.488849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bmqvx\""
Jul 02 14:22:03.488984 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.488852    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 14:22:03.495460 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.495442    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2wcb7"]
Jul 02 14:22:03.576330 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.576305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/466d2c69-6376-4e86-beed-c5a07c49f6da-cert\") pod \"kserve-controller-manager-7b9c77fff9-gphs5\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") " pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.576506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.576342    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khbc\" (UniqueName: \"kubernetes.io/projected/bad2322a-364c-4736-8077-7b40adf28e9a-kube-api-access-6khbc\") pod \"seaweedfs-64568bcd49-2wcb7\" (UID: \"bad2322a-364c-4736-8077-7b40adf28e9a\") " pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.576506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.576366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bad2322a-364c-4736-8077-7b40adf28e9a-data\") pod \"seaweedfs-64568bcd49-2wcb7\" (UID: \"bad2322a-364c-4736-8077-7b40adf28e9a\") " pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.576506 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.576387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkv6\" (UniqueName: \"kubernetes.io/projected/466d2c69-6376-4e86-beed-c5a07c49f6da-kube-api-access-vtkv6\") pod \"kserve-controller-manager-7b9c77fff9-gphs5\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") " pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.578662 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.578646    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/466d2c69-6376-4e86-beed-c5a07c49f6da-cert\") pod \"kserve-controller-manager-7b9c77fff9-gphs5\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") " pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.589697 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.589668    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkv6\" (UniqueName: \"kubernetes.io/projected/466d2c69-6376-4e86-beed-c5a07c49f6da-kube-api-access-vtkv6\") pod \"kserve-controller-manager-7b9c77fff9-gphs5\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") " pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.677694 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.677656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khbc\" (UniqueName: \"kubernetes.io/projected/bad2322a-364c-4736-8077-7b40adf28e9a-kube-api-access-6khbc\") pod \"seaweedfs-64568bcd49-2wcb7\" (UID: \"bad2322a-364c-4736-8077-7b40adf28e9a\") " pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.677694 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.677695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bad2322a-364c-4736-8077-7b40adf28e9a-data\") pod \"seaweedfs-64568bcd49-2wcb7\" (UID: \"bad2322a-364c-4736-8077-7b40adf28e9a\") " pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.678034 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.678017    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bad2322a-364c-4736-8077-7b40adf28e9a-data\") pod \"seaweedfs-64568bcd49-2wcb7\" (UID: \"bad2322a-364c-4736-8077-7b40adf28e9a\") " pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.710017 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.709986    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khbc\" (UniqueName: \"kubernetes.io/projected/bad2322a-364c-4736-8077-7b40adf28e9a-kube-api-access-6khbc\") pod \"seaweedfs-64568bcd49-2wcb7\" (UID: \"bad2322a-364c-4736-8077-7b40adf28e9a\") " pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.758030 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.757998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:03.794376 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.794340    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:03.907104 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.907078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-gphs5"]
Jul 02 14:22:03.909291 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:22:03.909264    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod466d2c69_6376_4e86_beed_c5a07c49f6da.slice/crio-a98953dad10ce7f891b52ac7e59c184468f99aa2b6ae054d8d1e85205011eb3b WatchSource:0}: Error finding container a98953dad10ce7f891b52ac7e59c184468f99aa2b6ae054d8d1e85205011eb3b: Status 404 returned error can't find the container with id a98953dad10ce7f891b52ac7e59c184468f99aa2b6ae054d8d1e85205011eb3b
Jul 02 14:22:03.938586 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:03.938550    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2wcb7"]
Jul 02 14:22:03.942197 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:22:03.942173    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad2322a_364c_4736_8077_7b40adf28e9a.slice/crio-06be0624ceca49731c6d52515fa02cf4a16fc97c55ff1cabf2b788df7762b8f0 WatchSource:0}: Error finding container 06be0624ceca49731c6d52515fa02cf4a16fc97c55ff1cabf2b788df7762b8f0: Status 404 returned error can't find the container with id 06be0624ceca49731c6d52515fa02cf4a16fc97c55ff1cabf2b788df7762b8f0
Jul 02 14:22:04.528455 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:04.528425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2wcb7" event={"ID":"bad2322a-364c-4736-8077-7b40adf28e9a","Type":"ContainerStarted","Data":"06be0624ceca49731c6d52515fa02cf4a16fc97c55ff1cabf2b788df7762b8f0"}
Jul 02 14:22:04.529499 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:04.529478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5" event={"ID":"466d2c69-6376-4e86-beed-c5a07c49f6da","Type":"ContainerStarted","Data":"a98953dad10ce7f891b52ac7e59c184468f99aa2b6ae054d8d1e85205011eb3b"}
Jul 02 14:22:08.546365 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:08.546334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2wcb7" event={"ID":"bad2322a-364c-4736-8077-7b40adf28e9a","Type":"ContainerStarted","Data":"a0d36e4d3992ca007723595c715a3c9ec079359ee6e40d2be43e23bfbc54b68b"}
Jul 02 14:22:08.546831 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:08.546422    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:08.547863 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:08.547829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5" event={"ID":"466d2c69-6376-4e86-beed-c5a07c49f6da","Type":"ContainerStarted","Data":"152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82"}
Jul 02 14:22:08.547985 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:08.547966    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:08.567920 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:08.567884    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2wcb7" podStartSLOduration=1.52303495 podStartE2EDuration="5.567870687s" totalImagesPullingTime="4.044835737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:22:08.567309145 +0000 UTC m=+390.822714490" watchObservedRunningTime="2026-07-02 14:22:08.567870687 +0000 UTC m=+390.823276030"
Jul 02 14:22:08.595265 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:08.595227    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5" podStartSLOduration=1.603164743 podStartE2EDuration="5.595216293s" totalImagesPullingTime="3.99205155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:22:08.594632632 +0000 UTC m=+390.850037972" watchObservedRunningTime="2026-07-02 14:22:08.595216293 +0000 UTC m=+390.850621691"
Jul 02 14:22:14.553367 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:14.553336    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2wcb7"
Jul 02 14:22:39.555821 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:39.555791    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:40.290152 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.290118    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-gphs5"]
Jul 02 14:22:40.290332 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.290291    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5" podUID="466d2c69-6376-4e86-beed-c5a07c49f6da" containerName="manager" containerID="cri-o://152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82" gracePeriod=10
Jul 02 14:22:40.317002 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.316975    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-b6mdl"]
Jul 02 14:22:40.319996 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.319978    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.333560 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.333535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-b6mdl"]
Jul 02 14:22:40.434978 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.434944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/464a6731-0562-4136-9441-84c9023d76a8-cert\") pod \"kserve-controller-manager-7b9c77fff9-b6mdl\" (UID: \"464a6731-0562-4136-9441-84c9023d76a8\") " pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.435246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.435221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hk4\" (UniqueName: \"kubernetes.io/projected/464a6731-0562-4136-9441-84c9023d76a8-kube-api-access-66hk4\") pod \"kserve-controller-manager-7b9c77fff9-b6mdl\" (UID: \"464a6731-0562-4136-9441-84c9023d76a8\") " pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.526461 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.526439    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:40.535978 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.535953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hk4\" (UniqueName: \"kubernetes.io/projected/464a6731-0562-4136-9441-84c9023d76a8-kube-api-access-66hk4\") pod \"kserve-controller-manager-7b9c77fff9-b6mdl\" (UID: \"464a6731-0562-4136-9441-84c9023d76a8\") " pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.536111 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.536032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/464a6731-0562-4136-9441-84c9023d76a8-cert\") pod \"kserve-controller-manager-7b9c77fff9-b6mdl\" (UID: \"464a6731-0562-4136-9441-84c9023d76a8\") " pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.538118 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.538098    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/464a6731-0562-4136-9441-84c9023d76a8-cert\") pod \"kserve-controller-manager-7b9c77fff9-b6mdl\" (UID: \"464a6731-0562-4136-9441-84c9023d76a8\") " pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.547905 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.547847    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hk4\" (UniqueName: \"kubernetes.io/projected/464a6731-0562-4136-9441-84c9023d76a8-kube-api-access-66hk4\") pod \"kserve-controller-manager-7b9c77fff9-b6mdl\" (UID: \"464a6731-0562-4136-9441-84c9023d76a8\") " pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.636857 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.636823    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtkv6\" (UniqueName: \"kubernetes.io/projected/466d2c69-6376-4e86-beed-c5a07c49f6da-kube-api-access-vtkv6\") pod \"466d2c69-6376-4e86-beed-c5a07c49f6da\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") "
Jul 02 14:22:40.637210 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.636928    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/466d2c69-6376-4e86-beed-c5a07c49f6da-cert\") pod \"466d2c69-6376-4e86-beed-c5a07c49f6da\" (UID: \"466d2c69-6376-4e86-beed-c5a07c49f6da\") "
Jul 02 14:22:40.638891 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.638866    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/466d2c69-6376-4e86-beed-c5a07c49f6da-kube-api-access-vtkv6" (OuterVolumeSpecName: "kube-api-access-vtkv6") pod "466d2c69-6376-4e86-beed-c5a07c49f6da" (UID: "466d2c69-6376-4e86-beed-c5a07c49f6da"). InnerVolumeSpecName "kube-api-access-vtkv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:22:40.638956 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.638907    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/466d2c69-6376-4e86-beed-c5a07c49f6da-cert" (OuterVolumeSpecName: "cert") pod "466d2c69-6376-4e86-beed-c5a07c49f6da" (UID: "466d2c69-6376-4e86-beed-c5a07c49f6da"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:22:40.651842 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.651820    2583 generic.go:356] "Generic (PLEG): container finished" podID="466d2c69-6376-4e86-beed-c5a07c49f6da" containerID="152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82" exitCode=0
Jul 02 14:22:40.651956 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.651870    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5"
Jul 02 14:22:40.651956 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.651879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5" event={"ID":"466d2c69-6376-4e86-beed-c5a07c49f6da","Type":"ContainerDied","Data":"152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82"}
Jul 02 14:22:40.651956 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.651930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-gphs5" event={"ID":"466d2c69-6376-4e86-beed-c5a07c49f6da","Type":"ContainerDied","Data":"a98953dad10ce7f891b52ac7e59c184468f99aa2b6ae054d8d1e85205011eb3b"}
Jul 02 14:22:40.651956 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.651954    2583 scope.go:117] "RemoveContainer" containerID="152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82"
Jul 02 14:22:40.660215 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.660193    2583 scope.go:117] "RemoveContainer" containerID="152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82"
Jul 02 14:22:40.660459 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:22:40.660439    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82\": container with ID starting with 152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82 not found: ID does not exist" containerID="152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82"
Jul 02 14:22:40.660535 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.660465    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82"} err="failed to get container status \"152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82\": rpc error: code = NotFound desc = could not find container \"152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82\": container with ID starting with 152cfb127774d4acf56a04d6360b0803a25df9e076a06db783bb5fdfb8a09c82 not found: ID does not exist"
Jul 02 14:22:40.667809 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.667789    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:40.673094 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.673077    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-gphs5"]
Jul 02 14:22:40.676822 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.676804    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-gphs5"]
Jul 02 14:22:40.738216 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.738188    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/466d2c69-6376-4e86-beed-c5a07c49f6da-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:22:40.738216 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.738209    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtkv6\" (UniqueName: \"kubernetes.io/projected/466d2c69-6376-4e86-beed-c5a07c49f6da-kube-api-access-vtkv6\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:22:40.791552 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:40.791486    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7b9c77fff9-b6mdl"]
Jul 02 14:22:40.793901 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:22:40.793873    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464a6731_0562_4136_9441_84c9023d76a8.slice/crio-1c2bec8b63c8585ba7cb3938d6f32031fcf2dde514685a62240a5eaa5292406b WatchSource:0}: Error finding container 1c2bec8b63c8585ba7cb3938d6f32031fcf2dde514685a62240a5eaa5292406b: Status 404 returned error can't find the container with id 1c2bec8b63c8585ba7cb3938d6f32031fcf2dde514685a62240a5eaa5292406b
Jul 02 14:22:41.656612 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:41.656573    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl" event={"ID":"464a6731-0562-4136-9441-84c9023d76a8","Type":"ContainerStarted","Data":"f165d7f1018146384994aa42531a2d1986ea16fb84f8480e2f0516ab98b8b665"}
Jul 02 14:22:41.657127 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:41.656633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl" event={"ID":"464a6731-0562-4136-9441-84c9023d76a8","Type":"ContainerStarted","Data":"1c2bec8b63c8585ba7cb3938d6f32031fcf2dde514685a62240a5eaa5292406b"}
Jul 02 14:22:41.657127 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:41.656684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:22:41.674543 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:41.674489    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl" podStartSLOduration=1.278010348 podStartE2EDuration="1.67447245s" totalImagesPullingTime="396.462102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:22:41.673622275 +0000 UTC m=+423.929027629" watchObservedRunningTime="2026-07-02 14:22:41.67447245 +0000 UTC m=+423.929877791"
Jul 02 14:22:42.148208 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:22:42.148178    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="466d2c69-6376-4e86-beed-c5a07c49f6da" path="/var/lib/kubelet/pods/466d2c69-6376-4e86-beed-c5a07c49f6da/volumes"
Jul 02 14:23:12.665761 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:12.665729    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7b9c77fff9-b6mdl"
Jul 02 14:23:13.789089 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.789063    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-6x7tz"]
Jul 02 14:23:13.789477 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.789298    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="466d2c69-6376-4e86-beed-c5a07c49f6da" containerName="manager"
Jul 02 14:23:13.789477 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.789309    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="466d2c69-6376-4e86-beed-c5a07c49f6da" containerName="manager"
Jul 02 14:23:13.789477 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.789352    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="466d2c69-6376-4e86-beed-c5a07c49f6da" containerName="manager"
Jul 02 14:23:13.792215 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.792198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:13.796206 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.796188    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4cg5s\""
Jul 02 14:23:13.796317 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.796231    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 02 14:23:13.806980 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.806959    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-6x7tz"]
Jul 02 14:23:13.876225 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.876194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wr4d\" (UniqueName: \"kubernetes.io/projected/6954789d-6e1d-4ac0-a625-832189b39a6e-kube-api-access-7wr4d\") pod \"model-serving-api-fd65d7d6-6x7tz\" (UID: \"6954789d-6e1d-4ac0-a625-832189b39a6e\") " pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:13.876380 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.876239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6954789d-6e1d-4ac0-a625-832189b39a6e-tls-certs\") pod \"model-serving-api-fd65d7d6-6x7tz\" (UID: \"6954789d-6e1d-4ac0-a625-832189b39a6e\") " pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:13.976543 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.976510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6954789d-6e1d-4ac0-a625-832189b39a6e-tls-certs\") pod \"model-serving-api-fd65d7d6-6x7tz\" (UID: \"6954789d-6e1d-4ac0-a625-832189b39a6e\") " pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:13.976691 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.976588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wr4d\" (UniqueName: \"kubernetes.io/projected/6954789d-6e1d-4ac0-a625-832189b39a6e-kube-api-access-7wr4d\") pod \"model-serving-api-fd65d7d6-6x7tz\" (UID: \"6954789d-6e1d-4ac0-a625-832189b39a6e\") " pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:13.979061 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.979039    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6954789d-6e1d-4ac0-a625-832189b39a6e-tls-certs\") pod \"model-serving-api-fd65d7d6-6x7tz\" (UID: \"6954789d-6e1d-4ac0-a625-832189b39a6e\") " pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:13.985797 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:13.985752    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wr4d\" (UniqueName: \"kubernetes.io/projected/6954789d-6e1d-4ac0-a625-832189b39a6e-kube-api-access-7wr4d\") pod \"model-serving-api-fd65d7d6-6x7tz\" (UID: \"6954789d-6e1d-4ac0-a625-832189b39a6e\") " pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:14.102474 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:14.102394    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:14.228293 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:14.228271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-6x7tz"]
Jul 02 14:23:14.230112 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:23:14.230084    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6954789d_6e1d_4ac0_a625_832189b39a6e.slice/crio-dbdb3729b6fb709296cb4b1a23c494c6c8aaa656e0e386505381384f4f7f7c16 WatchSource:0}: Error finding container dbdb3729b6fb709296cb4b1a23c494c6c8aaa656e0e386505381384f4f7f7c16: Status 404 returned error can't find the container with id dbdb3729b6fb709296cb4b1a23c494c6c8aaa656e0e386505381384f4f7f7c16
Jul 02 14:23:14.773293 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:14.773260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-6x7tz" event={"ID":"6954789d-6e1d-4ac0-a625-832189b39a6e","Type":"ContainerStarted","Data":"dbdb3729b6fb709296cb4b1a23c494c6c8aaa656e0e386505381384f4f7f7c16"}
Jul 02 14:23:17.784134 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:17.784102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-6x7tz" event={"ID":"6954789d-6e1d-4ac0-a625-832189b39a6e","Type":"ContainerStarted","Data":"5b3c50075436f9f3433395002dededa5eef47184d5d47a955e51bdcf513b0dc7"}
Jul 02 14:23:17.784505 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:17.784173    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:23:17.801776 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:17.801728    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-6x7tz" podStartSLOduration=1.8801892310000001 podStartE2EDuration="4.80171334s" totalImagesPullingTime="2.921524109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:23:17.801048341 +0000 UTC m=+460.056453698" watchObservedRunningTime="2026-07-02 14:23:17.80171334 +0000 UTC m=+460.057118682"
Jul 02 14:23:28.791228 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:23:28.791196    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-6x7tz"
Jul 02 14:25:01.949115 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.949082    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"]
Jul 02 14:25:01.952320 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.952300    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:01.954919 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.954901    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:25:01.955017 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.954928    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a23ad-kube-rbac-proxy-sar-config\""
Jul 02 14:25:01.955179 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.955166    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:25:01.955260 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.955240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-a23ad-predictor-serving-cert\""
Jul 02 14:25:01.956026 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.955986    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6qw4\""
Jul 02 14:25:01.956026 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.956005    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:25:01.956026 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.955989    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:25:01.962932 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:01.962914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"]
Jul 02 14:25:02.103611 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.103571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.103611 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.103615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf7xw\" (UniqueName: \"kubernetes.io/projected/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-api-access-gf7xw\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.103868 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.103641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.103868 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.103733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f759183-a980-4aa1-be4b-fd8f4f3b9222-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.103868 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.103793    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.204561 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.204479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f759183-a980-4aa1-be4b-fd8f4f3b9222-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.204561 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.204519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.204799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.204569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.204799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.204589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf7xw\" (UniqueName: \"kubernetes.io/projected/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-api-access-gf7xw\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.204799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.204606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.205026 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.205006    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.205345 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.205317    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.205452 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.205388    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.207039 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.207022    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f759183-a980-4aa1-be4b-fd8f4f3b9222-proxy-tls\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.213924 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.213904    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf7xw\" (UniqueName: \"kubernetes.io/projected/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-api-access-gf7xw\") pod \"isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.262874 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.262843    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:02.393495 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:02.393465    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"]
Jul 02 14:25:02.395468 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:25:02.395440    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f759183_a980_4aa1_be4b_fd8f4f3b9222.slice/crio-7262ae8545fe2873c08cb8bd83bfeb94195788a2d1cb83af66a02d45cd6e9f4b WatchSource:0}: Error finding container 7262ae8545fe2873c08cb8bd83bfeb94195788a2d1cb83af66a02d45cd6e9f4b: Status 404 returned error can't find the container with id 7262ae8545fe2873c08cb8bd83bfeb94195788a2d1cb83af66a02d45cd6e9f4b
Jul 02 14:25:03.145533 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:03.145499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerStarted","Data":"7262ae8545fe2873c08cb8bd83bfeb94195788a2d1cb83af66a02d45cd6e9f4b"}
Jul 02 14:25:07.162573 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:07.162539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerStarted","Data":"3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc"}
Jul 02 14:25:10.173904 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:10.173871    2583 generic.go:356] "Generic (PLEG): container finished" podID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerID="3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc" exitCode=0
Jul 02 14:25:10.174302 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:10.173936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerDied","Data":"3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc"}
Jul 02 14:25:25.241039 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:25.241002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerStarted","Data":"c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962"}
Jul 02 14:25:28.254708 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:28.254666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerStarted","Data":"0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7"}
Jul 02 14:25:31.267055 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:31.267018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerStarted","Data":"fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6"}
Jul 02 14:25:31.267479 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:31.267262    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:31.267479 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:31.267334    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:31.268492 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:31.268465    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:25:32.271598 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:32.271558    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:32.272100 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:32.271796    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:25:32.272866 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:32.272832    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:32.275713 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:32.275693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:25:32.301723 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:32.301679    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podStartSLOduration=3.013017845 podStartE2EDuration="31.301666069s" totalImagesPullingTime="24.478656152s" totalInitContainerRuntime="3.809992072s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:25:01 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:25:31.298918241 +0000 UTC m=+593.554323582" watchObservedRunningTime="2026-07-02 14:25:32.301666069 +0000 UTC m=+594.557071463"
Jul 02 14:25:33.275077 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:33.275040    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:25:33.275508 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:33.275382    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:34.281679 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:34.281631    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:25:34.282407 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:34.282378    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:38.110441 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:38.110409    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:25:38.111380 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:38.111359    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:25:44.282311 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:44.282263    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:25:44.282816 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:44.282761    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:25:54.282572 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:54.282529    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:25:54.283084 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:25:54.283025    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:26:04.282559 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:04.282512    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:26:04.285496 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:04.283046    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:26:14.282191 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:14.282138    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:26:14.282579 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:14.282547    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:26:24.282333 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:24.282285    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:26:24.282811 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:24.282789    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:26:34.282437 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:34.282389    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:26:34.282952 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:34.282836    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:26:36.144204 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:36.144164    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:26:36.144627 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:36.144357    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:26:46.148794 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:46.148746    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:26:46.149174 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:46.148866    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:26:57.159842 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.159807    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"]
Jul 02 14:26:57.160294 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.160117    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" containerID="cri-o://c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962" gracePeriod=30
Jul 02 14:26:57.160294 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.160167    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" containerID="cri-o://0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7" gracePeriod=30
Jul 02 14:26:57.160294 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.160268    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" containerID="cri-o://fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6" gracePeriod=30
Jul 02 14:26:57.210089 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.210054    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"]
Jul 02 14:26:57.213540 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.213522    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.216333 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.216310    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-042a3-kube-rbac-proxy-sar-config\""
Jul 02 14:26:57.216430 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.216310    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-042a3-predictor-serving-cert\""
Jul 02 14:26:57.223317 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.223299    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"]
Jul 02 14:26:57.272701 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.272641    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 02 14:26:57.361439 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.361408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.361620 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.361449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-proxy-tls\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.361620 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.361560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-cabundle-cert\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.361620 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.361608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.361753 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.361652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v9f8\" (UniqueName: \"kubernetes.io/projected/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-api-access-8v9f8\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.462874 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.462846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-cabundle-cert\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.463035 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.462887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.463035 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.462908    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v9f8\" (UniqueName: \"kubernetes.io/projected/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-api-access-8v9f8\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.463035 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.462954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.463035 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.462985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-proxy-tls\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.463398 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.463365    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.463655 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.463632    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-cabundle-cert\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.463705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.463632    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.465385 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.465367    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-proxy-tls\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.472292 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.472267    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v9f8\" (UniqueName: \"kubernetes.io/projected/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-api-access-8v9f8\") pod \"isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.524255 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.524229    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:26:57.575533 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.575499    2583 generic.go:356] "Generic (PLEG): container finished" podID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerID="0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7" exitCode=2
Jul 02 14:26:57.575695 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.575578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerDied","Data":"0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7"}
Jul 02 14:26:57.650028 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:57.650006    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"]
Jul 02 14:26:57.652345 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:26:57.652319    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb193ee_77ed_40f8_9b0c_43372ec20ca7.slice/crio-def6a15244c42d2582ff92d3a751fd3f706fdcc2a2e2196cc42efe8622ba2d19 WatchSource:0}: Error finding container def6a15244c42d2582ff92d3a751fd3f706fdcc2a2e2196cc42efe8622ba2d19: Status 404 returned error can't find the container with id def6a15244c42d2582ff92d3a751fd3f706fdcc2a2e2196cc42efe8622ba2d19
Jul 02 14:26:58.580964 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:58.580928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerStarted","Data":"1a9b6b98645b3bcebe77d125d9c49b0235c5937bcdcf6d47d1aacbdea462557a"}
Jul 02 14:26:58.581377 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:26:58.580989    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerStarted","Data":"def6a15244c42d2582ff92d3a751fd3f706fdcc2a2e2196cc42efe8622ba2d19"}
Jul 02 14:27:01.592500 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:01.592459    2583 generic.go:356] "Generic (PLEG): container finished" podID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerID="1a9b6b98645b3bcebe77d125d9c49b0235c5937bcdcf6d47d1aacbdea462557a" exitCode=0
Jul 02 14:27:01.592816 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:01.592533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerDied","Data":"1a9b6b98645b3bcebe77d125d9c49b0235c5937bcdcf6d47d1aacbdea462557a"}
Jul 02 14:27:02.272904 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.272859    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 02 14:27:02.602332 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.602248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerStarted","Data":"74eb18fd6367fd057c3d659cf9155fcacfeac649b66e58f4f808e1d1349c0ce3"}
Jul 02 14:27:02.602705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.602345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerStarted","Data":"719aac4c3c1f4de5e2f323a1afa3e627974d08051bc5f0676bf23acd79d4a67b"}
Jul 02 14:27:02.602705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.602624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:27:02.602853 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.602727    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:27:02.604353 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.604321    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:27:02.604735 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.604718    2583 generic.go:356] "Generic (PLEG): container finished" podID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerID="c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962" exitCode=0
Jul 02 14:27:02.604803 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.604785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerDied","Data":"c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962"}
Jul 02 14:27:02.623398 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:02.623360    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podStartSLOduration=2.072926239 podStartE2EDuration="5.623347776s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.550421537s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:27:02.622845366 +0000 UTC m=+684.878250707" watchObservedRunningTime="2026-07-02 14:27:02.623347776 +0000 UTC m=+684.878753113"
Jul 02 14:27:03.608260 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:03.608221    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:27:06.144131 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:06.144093    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:27:06.144626 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:06.144408    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:27:07.271950 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:07.271912    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 02 14:27:07.272366 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:07.272033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:27:08.613216 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:08.613190    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:27:08.613660 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:08.613637    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:27:12.272735 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:12.272688    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 02 14:27:16.144681 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:16.144636    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:27:16.145156 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:16.145007    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:27:17.272790 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:17.272728    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 02 14:27:18.614293 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:18.614253    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:27:22.272883 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:22.272839    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Jul 02 14:27:26.145091 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:26.145050    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Jul 02 14:27:26.145511 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:26.145301    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:27:26.150058 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:26.150037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:27:26.150198 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:26.150130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:27:27.308206 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.308183    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:27:27.382232 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382203    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f759183-a980-4aa1-be4b-fd8f4f3b9222-proxy-tls\") pod \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") "
Jul 02 14:27:27.382405 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf7xw\" (UniqueName: \"kubernetes.io/projected/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-api-access-gf7xw\") pod \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") "
Jul 02 14:27:27.382405 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382291    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kserve-provision-location\") pod \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") "
Jul 02 14:27:27.382405 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-rbac-proxy-sar-config\") pod \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") "
Jul 02 14:27:27.382405 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382352    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-cabundle-cert\") pod \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\" (UID: \"7f759183-a980-4aa1-be4b-fd8f4f3b9222\") "
Jul 02 14:27:27.382685 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382659    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f759183-a980-4aa1-be4b-fd8f4f3b9222" (UID: "7f759183-a980-4aa1-be4b-fd8f4f3b9222"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:27:27.382750 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382667    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f759183-a980-4aa1-be4b-fd8f4f3b9222" (UID: "7f759183-a980-4aa1-be4b-fd8f4f3b9222"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:27:27.382750 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.382717    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f759183-a980-4aa1-be4b-fd8f4f3b9222" (UID: "7f759183-a980-4aa1-be4b-fd8f4f3b9222"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:27:27.384403 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.384375    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-api-access-gf7xw" (OuterVolumeSpecName: "kube-api-access-gf7xw") pod "7f759183-a980-4aa1-be4b-fd8f4f3b9222" (UID: "7f759183-a980-4aa1-be4b-fd8f4f3b9222"). InnerVolumeSpecName "kube-api-access-gf7xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:27:27.384403 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.384388    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f759183-a980-4aa1-be4b-fd8f4f3b9222-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f759183-a980-4aa1-be4b-fd8f4f3b9222" (UID: "7f759183-a980-4aa1-be4b-fd8f4f3b9222"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:27:27.483530 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.483498    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf7xw\" (UniqueName: \"kubernetes.io/projected/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-api-access-gf7xw\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:27:27.483530 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.483524    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:27:27.483530 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.483533    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:27:27.483741 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.483542    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f759183-a980-4aa1-be4b-fd8f4f3b9222-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:27:27.483741 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.483551    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f759183-a980-4aa1-be4b-fd8f4f3b9222-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:27:27.698202 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.698173    2583 generic.go:356] "Generic (PLEG): container finished" podID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerID="fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6" exitCode=0
Jul 02 14:27:27.698352 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.698241    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"
Jul 02 14:27:27.698352 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.698254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerDied","Data":"fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6"}
Jul 02 14:27:27.698352 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.698331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" event={"ID":"7f759183-a980-4aa1-be4b-fd8f4f3b9222","Type":"ContainerDied","Data":"7262ae8545fe2873c08cb8bd83bfeb94195788a2d1cb83af66a02d45cd6e9f4b"}
Jul 02 14:27:27.698454 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.698375    2583 scope.go:117] "RemoveContainer" containerID="fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6"
Jul 02 14:27:27.706901 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.706885    2583 scope.go:117] "RemoveContainer" containerID="0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7"
Jul 02 14:27:27.714435 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.714417    2583 scope.go:117] "RemoveContainer" containerID="c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962"
Jul 02 14:27:27.721449 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.721426    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"]
Jul 02 14:27:27.721683 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.721670    2583 scope.go:117] "RemoveContainer" containerID="3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc"
Jul 02 14:27:27.725383 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.725364    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn"]
Jul 02 14:27:27.728669 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.728641    2583 scope.go:117] "RemoveContainer" containerID="fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6"
Jul 02 14:27:27.728937 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:27:27.728916    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6\": container with ID starting with fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6 not found: ID does not exist" containerID="fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6"
Jul 02 14:27:27.729017 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.728944    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6"} err="failed to get container status \"fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6\": rpc error: code = NotFound desc = could not find container \"fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6\": container with ID starting with fb7deccc5a926cd3509b85cd75e239a40aae718b0d34938e4c6db4567fe8caf6 not found: ID does not exist"
Jul 02 14:27:27.729017 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.728960    2583 scope.go:117] "RemoveContainer" containerID="0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7"
Jul 02 14:27:27.729188 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:27:27.729174    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7\": container with ID starting with 0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7 not found: ID does not exist" containerID="0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7"
Jul 02 14:27:27.729236 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.729191    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7"} err="failed to get container status \"0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7\": rpc error: code = NotFound desc = could not find container \"0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7\": container with ID starting with 0c34248d7944cb167bc4c13a5dd72e3499a852cc0442810af217a4ec078370e7 not found: ID does not exist"
Jul 02 14:27:27.729236 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.729203    2583 scope.go:117] "RemoveContainer" containerID="c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962"
Jul 02 14:27:27.729433 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:27:27.729415    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962\": container with ID starting with c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962 not found: ID does not exist" containerID="c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962"
Jul 02 14:27:27.729473 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.729438    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962"} err="failed to get container status \"c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962\": rpc error: code = NotFound desc = could not find container \"c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962\": container with ID starting with c5cd0d81dacd3e9e374549844898e357b105535549a3bc8bed3c939ce04bc962 not found: ID does not exist"
Jul 02 14:27:27.729473 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.729454    2583 scope.go:117] "RemoveContainer" containerID="3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc"
Jul 02 14:27:27.729653 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:27:27.729636    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc\": container with ID starting with 3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc not found: ID does not exist" containerID="3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc"
Jul 02 14:27:27.729700 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:27.729655    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc"} err="failed to get container status \"3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc\": rpc error: code = NotFound desc = could not find container \"3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc\": container with ID starting with 3ca7a0be8baa0a56b1bb4e6130b0fc0d24483be8d8c25178e6747e5cb42512cc not found: ID does not exist"
Jul 02 14:27:28.148395 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:28.148362    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" path="/var/lib/kubelet/pods/7f759183-a980-4aa1-be4b-fd8f4f3b9222/volumes"
Jul 02 14:27:28.272543 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:28.272504    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-a23ad-predictor-9694547ff-7dpfn" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 02 14:27:28.613652 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:28.613618    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:27:38.614597 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:38.614556    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:27:48.614500 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:48.614461    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:27:58.614481 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:27:58.614442    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:28:08.614561 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:08.614523    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:28:18.614969 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:18.614933    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:28:47.526555 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.526523    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"]
Jul 02 14:28:47.527058 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.526820    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" containerID="cri-o://719aac4c3c1f4de5e2f323a1afa3e627974d08051bc5f0676bf23acd79d4a67b" gracePeriod=30
Jul 02 14:28:47.527058 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.526903    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kube-rbac-proxy" containerID="cri-o://74eb18fd6367fd057c3d659cf9155fcacfeac649b66e58f4f808e1d1349c0ce3" gracePeriod=30
Jul 02 14:28:47.529811 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.529787    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"]
Jul 02 14:28:47.530207 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530191    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container"
Jul 02 14:28:47.530291 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530209    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container"
Jul 02 14:28:47.530291 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530228    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent"
Jul 02 14:28:47.530291 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530237    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent"
Jul 02 14:28:47.530291 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530256    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="storage-initializer"
Jul 02 14:28:47.530291 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530264    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="storage-initializer"
Jul 02 14:28:47.530291 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530276    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy"
Jul 02 14:28:47.530291 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530285    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy"
Jul 02 14:28:47.530625 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530344    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kserve-container"
Jul 02 14:28:47.530625 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530359    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="agent"
Jul 02 14:28:47.530625 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.530368    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f759183-a980-4aa1-be4b-fd8f4f3b9222" containerName="kube-rbac-proxy"
Jul 02 14:28:47.534028 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.533988    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.538427 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.538396    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-130af-predictor-serving-cert\""
Jul 02 14:28:47.538427 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.538412    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-130af-kube-rbac-proxy-sar-config\""
Jul 02 14:28:47.541854 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.541834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.541966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.541862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.541966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.541881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmfk\" (UniqueName: \"kubernetes.io/projected/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-api-access-bpmfk\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.542078 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.541965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.542078 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.542005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.546263 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.546245    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"]
Jul 02 14:28:47.642847 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.642813    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.642847 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.642858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.643109 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.642915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.643109 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.642944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.643109 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.642974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmfk\" (UniqueName: \"kubernetes.io/projected/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-api-access-bpmfk\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.643109 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:28:47.643034    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-130af-predictor-serving-cert" not found
Jul 02 14:28:47.643394 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:28:47.643120    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls podName:5a623e37-311c-4b1d-a32e-d7c8ea7db99f nodeName:}" failed. No retries permitted until 2026-07-02 14:28:48.143098286 +0000 UTC m=+790.398503607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" (UID: "5a623e37-311c-4b1d-a32e-d7c8ea7db99f") : secret "isvc-sklearn-graph-raw-hpa-130af-predictor-serving-cert" not found
Jul 02 14:28:47.643394 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.643323    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.643528 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.643509    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.643564 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.643530    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.652233 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.652210    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmfk\" (UniqueName: \"kubernetes.io/projected/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-api-access-bpmfk\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:47.969156 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.969127    2583 generic.go:356] "Generic (PLEG): container finished" podID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerID="74eb18fd6367fd057c3d659cf9155fcacfeac649b66e58f4f808e1d1349c0ce3" exitCode=2
Jul 02 14:28:47.969296 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:47.969197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerDied","Data":"74eb18fd6367fd057c3d659cf9155fcacfeac649b66e58f4f808e1d1349c0ce3"}
Jul 02 14:28:48.147078 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.147041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:48.149309 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.149286    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:48.156214 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.156194    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:48.282723 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.282701    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"]
Jul 02 14:28:48.283990 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:28:48.283966    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a623e37_311c_4b1d_a32e_d7c8ea7db99f.slice/crio-b13a74f650450e3320f372ccbf1e0f9819afbd23ebbe1119240b8753f73d4e9b WatchSource:0}: Error finding container b13a74f650450e3320f372ccbf1e0f9819afbd23ebbe1119240b8753f73d4e9b: Status 404 returned error can't find the container with id b13a74f650450e3320f372ccbf1e0f9819afbd23ebbe1119240b8753f73d4e9b
Jul 02 14:28:48.608809 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.608706    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Jul 02 14:28:48.614075 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.614031    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Jul 02 14:28:48.974383 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.974343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerStarted","Data":"6e074ae00c7a84e3765d637ea22bff1a6ec8102b41edfab69e226ed4169b214d"}
Jul 02 14:28:48.974557 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:48.974415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerStarted","Data":"b13a74f650450e3320f372ccbf1e0f9819afbd23ebbe1119240b8753f73d4e9b"}
Jul 02 14:28:51.988122 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:51.988085    2583 generic.go:356] "Generic (PLEG): container finished" podID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerID="719aac4c3c1f4de5e2f323a1afa3e627974d08051bc5f0676bf23acd79d4a67b" exitCode=0
Jul 02 14:28:51.988489 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:51.988138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerDied","Data":"719aac4c3c1f4de5e2f323a1afa3e627974d08051bc5f0676bf23acd79d4a67b"}
Jul 02 14:28:52.071621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.071601    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:28:52.174364 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174339    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-cabundle-cert\") pod \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") "
Jul 02 14:28:52.174468 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174369    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-proxy-tls\") pod \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") "
Jul 02 14:28:52.174468 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174389    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kserve-provision-location\") pod \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") "
Jul 02 14:28:52.174468 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174433    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v9f8\" (UniqueName: \"kubernetes.io/projected/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-api-access-8v9f8\") pod \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") "
Jul 02 14:28:52.174468 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-rbac-proxy-sar-config\") pod \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\" (UID: \"cbb193ee-77ed-40f8-9b0c-43372ec20ca7\") "
Jul 02 14:28:52.174783 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174717    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbb193ee-77ed-40f8-9b0c-43372ec20ca7" (UID: "cbb193ee-77ed-40f8-9b0c-43372ec20ca7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:28:52.174886 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174798    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbb193ee-77ed-40f8-9b0c-43372ec20ca7" (UID: "cbb193ee-77ed-40f8-9b0c-43372ec20ca7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:28:52.174886 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.174842    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cbb193ee-77ed-40f8-9b0c-43372ec20ca7" (UID: "cbb193ee-77ed-40f8-9b0c-43372ec20ca7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:28:52.176222 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.176206    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbb193ee-77ed-40f8-9b0c-43372ec20ca7" (UID: "cbb193ee-77ed-40f8-9b0c-43372ec20ca7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:28:52.176357 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.176336    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-api-access-8v9f8" (OuterVolumeSpecName: "kube-api-access-8v9f8") pod "cbb193ee-77ed-40f8-9b0c-43372ec20ca7" (UID: "cbb193ee-77ed-40f8-9b0c-43372ec20ca7"). InnerVolumeSpecName "kube-api-access-8v9f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:28:52.275903 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.275869    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v9f8\" (UniqueName: \"kubernetes.io/projected/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-api-access-8v9f8\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:28:52.275903 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.275898    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:28:52.275903 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.275909    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:28:52.276121 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.275918    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:28:52.276121 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.275927    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbb193ee-77ed-40f8-9b0c-43372ec20ca7-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:28:52.994070 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.994046    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"
Jul 02 14:28:52.994520 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.994041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt" event={"ID":"cbb193ee-77ed-40f8-9b0c-43372ec20ca7","Type":"ContainerDied","Data":"def6a15244c42d2582ff92d3a751fd3f706fdcc2a2e2196cc42efe8622ba2d19"}
Jul 02 14:28:52.994520 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.994174    2583 scope.go:117] "RemoveContainer" containerID="74eb18fd6367fd057c3d659cf9155fcacfeac649b66e58f4f808e1d1349c0ce3"
Jul 02 14:28:52.995664 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.995642    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerID="6e074ae00c7a84e3765d637ea22bff1a6ec8102b41edfab69e226ed4169b214d" exitCode=0
Jul 02 14:28:52.995753 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:52.995687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerDied","Data":"6e074ae00c7a84e3765d637ea22bff1a6ec8102b41edfab69e226ed4169b214d"}
Jul 02 14:28:53.002748 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:53.002422    2583 scope.go:117] "RemoveContainer" containerID="719aac4c3c1f4de5e2f323a1afa3e627974d08051bc5f0676bf23acd79d4a67b"
Jul 02 14:28:53.009928 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:53.009912    2583 scope.go:117] "RemoveContainer" containerID="1a9b6b98645b3bcebe77d125d9c49b0235c5937bcdcf6d47d1aacbdea462557a"
Jul 02 14:28:53.031427 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:53.031408    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"]
Jul 02 14:28:53.033815 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:53.033797    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-042a3-predictor-c6d85fb5f-r7hdt"]
Jul 02 14:28:54.002349 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:54.002320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerStarted","Data":"e5e970fbc1479955663338d498e2e87ff856c44b01d6e77ca5cc562f23f4ad2f"}
Jul 02 14:28:54.002717 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:54.002384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerStarted","Data":"bfd105a44aef23b214a4f8e36e18e2076b929591499a15ee2a33e0d83f49d02a"}
Jul 02 14:28:54.002717 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:54.002573    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:54.027294 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:54.027227    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podStartSLOduration=3.21943034 podStartE2EDuration="7.027211462s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.807781122s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:28:54.02625812 +0000 UTC m=+796.281663453" watchObservedRunningTime="2026-07-02 14:28:54.027211462 +0000 UTC m=+796.282616802"
Jul 02 14:28:54.148963 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:54.148928    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" path="/var/lib/kubelet/pods/cbb193ee-77ed-40f8-9b0c-43372ec20ca7/volumes"
Jul 02 14:28:55.005486 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:55.005457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:28:55.006866 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:55.006839    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:28:56.008829 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:28:56.008794    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:29:01.014186 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:29:01.014158    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:29:01.014678 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:29:01.014654    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:29:11.014995 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:29:11.014956    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:29:21.014858 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:29:21.014820    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:29:31.014680 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:29:31.014638    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:29:41.015347 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:29:41.015306    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:29:51.015373 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:29:51.015331    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:30:01.015362 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:01.015320    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:30:09.143926 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:09.143889    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:30:37.796577 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796496    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"]
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796777    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container"
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796788    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container"
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796807    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="storage-initializer"
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796813    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="storage-initializer"
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796822    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kube-rbac-proxy"
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796828    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kube-rbac-proxy"
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796879    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kserve-container"
Jul 02 14:30:37.796983 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.796887    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb193ee-77ed-40f8-9b0c-43372ec20ca7" containerName="kube-rbac-proxy"
Jul 02 14:30:37.800494 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.800475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.803039 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.803015    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-f89aa-kube-rbac-proxy-sar-config\""
Jul 02 14:30:37.803147 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.803038    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-f89aa-predictor-serving-cert\""
Jul 02 14:30:37.810922 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.810901    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"]
Jul 02 14:30:37.836411 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.836386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2q2\" (UniqueName: \"kubernetes.io/projected/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-api-access-xb2q2\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.836546 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.836471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.836546 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.836514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.867129 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.867106    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"]
Jul 02 14:30:37.867366 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.867320    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" containerID="cri-o://bfd105a44aef23b214a4f8e36e18e2076b929591499a15ee2a33e0d83f49d02a" gracePeriod=30
Jul 02 14:30:37.867437 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.867362    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kube-rbac-proxy" containerID="cri-o://e5e970fbc1479955663338d498e2e87ff856c44b01d6e77ca5cc562f23f4ad2f" gracePeriod=30
Jul 02 14:30:37.937990 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.937956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2q2\" (UniqueName: \"kubernetes.io/projected/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-api-access-xb2q2\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.937990 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.937996    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.938188 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.938016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.938188 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:30:37.938141    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-serving-cert: secret "message-dumper-raw-f89aa-predictor-serving-cert" not found
Jul 02 14:30:37.938267 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:30:37.938206    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls podName:d24544ce-6fc5-4cd7-a1fb-1f7fc65702de nodeName:}" failed. No retries permitted until 2026-07-02 14:30:38.438188173 +0000 UTC m=+900.693593512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls") pod "message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" (UID: "d24544ce-6fc5-4cd7-a1fb-1f7fc65702de") : secret "message-dumper-raw-f89aa-predictor-serving-cert" not found
Jul 02 14:30:37.938631 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.938611    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:37.947067 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:37.947043    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2q2\" (UniqueName: \"kubernetes.io/projected/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-api-access-xb2q2\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:38.132093 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.132015    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:30:38.132093 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.132063    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:30:38.365706 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.365671    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerID="e5e970fbc1479955663338d498e2e87ff856c44b01d6e77ca5cc562f23f4ad2f" exitCode=2
Jul 02 14:30:38.365706 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.365706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerDied","Data":"e5e970fbc1479955663338d498e2e87ff856c44b01d6e77ca5cc562f23f4ad2f"}
Jul 02 14:30:38.442362 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.442278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:38.444577 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.444554    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls\") pod \"message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") " pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:38.728799 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.728743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:38.856668 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.856639    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"]
Jul 02 14:30:38.860168 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:30:38.860139    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24544ce_6fc5_4cd7_a1fb_1f7fc65702de.slice/crio-f003fae550d48e711f90aa91e8abb39f8c743c13268997be0dc2058bba944a76 WatchSource:0}: Error finding container f003fae550d48e711f90aa91e8abb39f8c743c13268997be0dc2058bba944a76: Status 404 returned error can't find the container with id f003fae550d48e711f90aa91e8abb39f8c743c13268997be0dc2058bba944a76
Jul 02 14:30:38.862080 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:38.862061    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 14:30:39.144244 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:39.144144    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 02 14:30:39.370271 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:39.370234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" event={"ID":"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de","Type":"ContainerStarted","Data":"f003fae550d48e711f90aa91e8abb39f8c743c13268997be0dc2058bba944a76"}
Jul 02 14:30:40.376570 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:40.376526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" event={"ID":"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de","Type":"ContainerStarted","Data":"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"}
Jul 02 14:30:40.377050 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:40.376606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" event={"ID":"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de","Type":"ContainerStarted","Data":"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"}
Jul 02 14:30:40.377050 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:40.376643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:40.377050 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:40.376673    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:40.378794 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:40.378712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:40.399130 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:40.399087    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" podStartSLOduration=2.13281527 podStartE2EDuration="3.399072982s" totalImagesPullingTime="1.266257712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:30:40.397910999 +0000 UTC m=+902.653316413" watchObservedRunningTime="2026-07-02 14:30:40.399072982 +0000 UTC m=+902.654478387"
Jul 02 14:30:41.010008 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:41.009964    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 02 14:30:42.386066 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.386038    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerID="bfd105a44aef23b214a4f8e36e18e2076b929591499a15ee2a33e0d83f49d02a" exitCode=0
Jul 02 14:30:42.386394 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.386110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerDied","Data":"bfd105a44aef23b214a4f8e36e18e2076b929591499a15ee2a33e0d83f49d02a"}
Jul 02 14:30:42.515408 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.515385    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:30:42.575998 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.575915    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls\") pod \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") "
Jul 02 14:30:42.575998 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.575973    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-rbac-proxy-sar-config\") pod \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") "
Jul 02 14:30:42.575998 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.575999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-cabundle-cert\") pod \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") "
Jul 02 14:30:42.576532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.576105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpmfk\" (UniqueName: \"kubernetes.io/projected/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-api-access-bpmfk\") pod \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") "
Jul 02 14:30:42.576532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.576140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kserve-provision-location\") pod \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\" (UID: \"5a623e37-311c-4b1d-a32e-d7c8ea7db99f\") "
Jul 02 14:30:42.576532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.576332    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a623e37-311c-4b1d-a32e-d7c8ea7db99f" (UID: "5a623e37-311c-4b1d-a32e-d7c8ea7db99f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:30:42.576532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.576453    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a623e37-311c-4b1d-a32e-d7c8ea7db99f" (UID: "5a623e37-311c-4b1d-a32e-d7c8ea7db99f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:30:42.576532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.576500    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a623e37-311c-4b1d-a32e-d7c8ea7db99f" (UID: "5a623e37-311c-4b1d-a32e-d7c8ea7db99f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:30:42.578095 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.578066    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a623e37-311c-4b1d-a32e-d7c8ea7db99f" (UID: "5a623e37-311c-4b1d-a32e-d7c8ea7db99f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:30:42.578095 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.578076    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-api-access-bpmfk" (OuterVolumeSpecName: "kube-api-access-bpmfk") pod "5a623e37-311c-4b1d-a32e-d7c8ea7db99f" (UID: "5a623e37-311c-4b1d-a32e-d7c8ea7db99f"). InnerVolumeSpecName "kube-api-access-bpmfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:30:42.676782 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.676747    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:30:42.676782 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.676784    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:30:42.676982 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.676793    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:30:42.676982 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.676802    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpmfk\" (UniqueName: \"kubernetes.io/projected/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kube-api-access-bpmfk\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:30:42.676982 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:42.676810    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a623e37-311c-4b1d-a32e-d7c8ea7db99f-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:30:43.391455 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:43.391417    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk" event={"ID":"5a623e37-311c-4b1d-a32e-d7c8ea7db99f","Type":"ContainerDied","Data":"b13a74f650450e3320f372ccbf1e0f9819afbd23ebbe1119240b8753f73d4e9b"}
Jul 02 14:30:43.391845 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:43.391480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"
Jul 02 14:30:43.391845 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:43.391509    2583 scope.go:117] "RemoveContainer" containerID="e5e970fbc1479955663338d498e2e87ff856c44b01d6e77ca5cc562f23f4ad2f"
Jul 02 14:30:43.399260 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:43.399239    2583 scope.go:117] "RemoveContainer" containerID="bfd105a44aef23b214a4f8e36e18e2076b929591499a15ee2a33e0d83f49d02a"
Jul 02 14:30:43.406417 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:43.406398    2583 scope.go:117] "RemoveContainer" containerID="6e074ae00c7a84e3765d637ea22bff1a6ec8102b41edfab69e226ed4169b214d"
Jul 02 14:30:43.415409 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:43.415389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"]
Jul 02 14:30:43.418377 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:43.418357    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-130af-predictor-6bd56fd467-6wprk"]
Jul 02 14:30:44.149076 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:44.149039    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" path="/var/lib/kubelet/pods/5a623e37-311c-4b1d-a32e-d7c8ea7db99f/volumes"
Jul 02 14:30:47.391708 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.391679    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:30:47.829857 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.829818    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"]
Jul 02 14:30:47.830097 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830085    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container"
Jul 02 14:30:47.830140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830098    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container"
Jul 02 14:30:47.830140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830119    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kube-rbac-proxy"
Jul 02 14:30:47.830140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830124    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kube-rbac-proxy"
Jul 02 14:30:47.830140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830131    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="storage-initializer"
Jul 02 14:30:47.830140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830137    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="storage-initializer"
Jul 02 14:30:47.830302 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830199    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kube-rbac-proxy"
Jul 02 14:30:47.830302 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.830207    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a623e37-311c-4b1d-a32e-d7c8ea7db99f" containerName="kserve-container"
Jul 02 14:30:47.834860 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.834839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:47.837568 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.837542    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-f89aa-predictor-serving-cert\""
Jul 02 14:30:47.837689 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.837593    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-f89aa-kube-rbac-proxy-sar-config\""
Jul 02 14:30:47.837689 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.837597    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:30:47.837689 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.837678    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:30:47.845445 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.845422    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"]
Jul 02 14:30:47.913377 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.913336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:47.913556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.913392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-cabundle-cert\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:47.913556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.913445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kserve-provision-location\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:47.913556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.913524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-proxy-tls\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:47.913672 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:47.913590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvmb\" (UniqueName: \"kubernetes.io/projected/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-api-access-vkvmb\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.014306 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.014271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.014306 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.014308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-cabundle-cert\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.014573 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.014332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kserve-provision-location\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.014573 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.014359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-proxy-tls\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.014573 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.014394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvmb\" (UniqueName: \"kubernetes.io/projected/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-api-access-vkvmb\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.014849 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.014820    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kserve-provision-location\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.015154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.015133    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.015200 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.015133    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-cabundle-cert\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.016894 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.016878    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-proxy-tls\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.024709 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.024687    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvmb\" (UniqueName: \"kubernetes.io/projected/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-api-access-vkvmb\") pod \"isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") " pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.145072 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.144980    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:48.275574 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.275539    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"]
Jul 02 14:30:48.278725 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:30:48.278695    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fee74b3_e784_45c0_82bb_c2cfe9c68c68.slice/crio-66fcbe53eef256a8d3e88a0ff02147154c5a84b3fafeef1c6aa07be76f64202f WatchSource:0}: Error finding container 66fcbe53eef256a8d3e88a0ff02147154c5a84b3fafeef1c6aa07be76f64202f: Status 404 returned error can't find the container with id 66fcbe53eef256a8d3e88a0ff02147154c5a84b3fafeef1c6aa07be76f64202f
Jul 02 14:30:48.415665 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.415567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerStarted","Data":"b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a"}
Jul 02 14:30:48.416196 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:48.415674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerStarted","Data":"66fcbe53eef256a8d3e88a0ff02147154c5a84b3fafeef1c6aa07be76f64202f"}
Jul 02 14:30:52.431227 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:52.431193    2583 generic.go:356] "Generic (PLEG): container finished" podID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerID="b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a" exitCode=0
Jul 02 14:30:52.431596 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:52.431266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerDied","Data":"b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a"}
Jul 02 14:30:53.437534 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:53.437500    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerStarted","Data":"3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba"}
Jul 02 14:30:53.438036 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:53.437575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerStarted","Data":"1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767"}
Jul 02 14:30:53.438036 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:53.437610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerStarted","Data":"ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0"}
Jul 02 14:30:53.438036 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:53.437992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:53.463147 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:53.463104    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podStartSLOduration=2.964029968 podStartE2EDuration="6.46309216s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.499062192s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:30:53.461738668 +0000 UTC m=+915.717144032" watchObservedRunningTime="2026-07-02 14:30:53.46309216 +0000 UTC m=+915.718497501"
Jul 02 14:30:54.440981 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:54.440941    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:54.441354 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:54.441019    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:30:54.442411 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:54.442384    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:30:54.443055 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:54.443032    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:30:55.444690 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:55.444645    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:30:55.445181 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:30:55.445038    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:31:00.449004 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:00.448978    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:31:00.449673 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:00.449640    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:31:00.449848 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:00.449821    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:31:10.450041 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:10.449993    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:31:10.452555 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:10.450477    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:31:20.449720 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:20.449679    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:31:20.450228 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:20.450205    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:31:30.449733 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:30.449677    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:31:30.450160 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:30.450130    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:31:40.449669 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:40.449631    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:31:40.450097 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:40.450059    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:31:50.450459 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:50.450421    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:31:50.451019 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:31:50.450967    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:32:00.449886 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:00.449844    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:32:00.450342 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:00.450273    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:32:10.450221 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:10.450192    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:32:10.450613 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:10.450385    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:32:22.859008 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:22.858978    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg_d24544ce-6fc5-4cd7-a1fb-1f7fc65702de/kserve-container/0.log"
Jul 02 14:32:23.065946 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.065915    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"]
Jul 02 14:32:23.066261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.066232    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" containerID="cri-o://ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0" gracePeriod=30
Jul 02 14:32:23.066508 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.066288    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" containerID="cri-o://3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba" gracePeriod=30
Jul 02 14:32:23.066633 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.066291    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy" containerID="cri-o://1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767" gracePeriod=30
Jul 02 14:32:23.146328 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.146254    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"]
Jul 02 14:32:23.150018 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.149997    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.152569 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.152547    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a9dfc-predictor-serving-cert\""
Jul 02 14:32:23.152680 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.152575    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-a9dfc-kube-rbac-proxy-sar-config\""
Jul 02 14:32:23.161226 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.161200    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"]
Jul 02 14:32:23.211027 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.210998    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"]
Jul 02 14:32:23.211221 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.211193    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kserve-container" containerID="cri-o://788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9" gracePeriod=30
Jul 02 14:32:23.211350 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.211229    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kube-rbac-proxy" containerID="cri-o://7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4" gracePeriod=30
Jul 02 14:32:23.228808 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.228757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.228903 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.228834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.228903 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.228863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlr89\" (UniqueName: \"kubernetes.io/projected/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-api-access-nlr89\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.228903 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.228881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90c7b87-d358-4e52-8f9c-f66207d43db1-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.229039 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.228938    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330130 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330220 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330220 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlr89\" (UniqueName: \"kubernetes.io/projected/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-api-access-nlr89\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330220 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90c7b87-d358-4e52-8f9c-f66207d43db1-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330370 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330370 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:23.330288    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-serving-cert: secret "isvc-sklearn-scale-raw-a9dfc-predictor-serving-cert" not found
Jul 02 14:32:23.330370 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:23.330360    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls podName:f90c7b87-d358-4e52-8f9c-f66207d43db1 nodeName:}" failed. No retries permitted until 2026-07-02 14:32:23.830336646 +0000 UTC m=+1006.085741965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls") pod "isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" (UID: "f90c7b87-d358-4e52-8f9c-f66207d43db1") : secret "isvc-sklearn-scale-raw-a9dfc-predictor-serving-cert" not found
Jul 02 14:32:23.330756 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330735    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90c7b87-d358-4e52-8f9c-f66207d43db1-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330899 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330879    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-cabundle-cert\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.330941 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.330916    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.339401 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.339378    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlr89\" (UniqueName: \"kubernetes.io/projected/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-api-access-nlr89\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.449411 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.449390    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:32:23.531644 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.531614    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls\") pod \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") "
Jul 02 14:32:23.531816 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.531656    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-rbac-proxy-sar-config\") pod \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") "
Jul 02 14:32:23.531816 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.531675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb2q2\" (UniqueName: \"kubernetes.io/projected/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-api-access-xb2q2\") pod \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\" (UID: \"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de\") "
Jul 02 14:32:23.532031 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.532008    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" (UID: "d24544ce-6fc5-4cd7-a1fb-1f7fc65702de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:32:23.533711 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.533680    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" (UID: "d24544ce-6fc5-4cd7-a1fb-1f7fc65702de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:32:23.533835 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.533719    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-api-access-xb2q2" (OuterVolumeSpecName: "kube-api-access-xb2q2") pod "d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" (UID: "d24544ce-6fc5-4cd7-a1fb-1f7fc65702de"). InnerVolumeSpecName "kube-api-access-xb2q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:32:23.632456 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.632423    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:23.632456 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.632448    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:23.632456 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.632458    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb2q2\" (UniqueName: \"kubernetes.io/projected/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de-kube-api-access-xb2q2\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:23.762272 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.762236    2583 generic.go:356] "Generic (PLEG): container finished" podID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerID="7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4" exitCode=2
Jul 02 14:32:23.762272 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.762262    2583 generic.go:356] "Generic (PLEG): container finished" podID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerID="788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9" exitCode=2
Jul 02 14:32:23.762509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.762293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"
Jul 02 14:32:23.762509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.762324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" event={"ID":"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de","Type":"ContainerDied","Data":"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"}
Jul 02 14:32:23.762509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.762383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" event={"ID":"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de","Type":"ContainerDied","Data":"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"}
Jul 02 14:32:23.762509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.762406    2583 scope.go:117] "RemoveContainer" containerID="7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"
Jul 02 14:32:23.762509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.762413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg" event={"ID":"d24544ce-6fc5-4cd7-a1fb-1f7fc65702de","Type":"ContainerDied","Data":"f003fae550d48e711f90aa91e8abb39f8c743c13268997be0dc2058bba944a76"}
Jul 02 14:32:23.764933 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.764912    2583 generic.go:356] "Generic (PLEG): container finished" podID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerID="1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767" exitCode=2
Jul 02 14:32:23.765068 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.764980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerDied","Data":"1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767"}
Jul 02 14:32:23.770397 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.770382    2583 scope.go:117] "RemoveContainer" containerID="788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"
Jul 02 14:32:23.777626 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.777609    2583 scope.go:117] "RemoveContainer" containerID="7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"
Jul 02 14:32:23.777887 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:23.777867    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4\": container with ID starting with 7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4 not found: ID does not exist" containerID="7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"
Jul 02 14:32:23.777945 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.777895    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"} err="failed to get container status \"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4\": rpc error: code = NotFound desc = could not find container \"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4\": container with ID starting with 7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4 not found: ID does not exist"
Jul 02 14:32:23.777945 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.777911    2583 scope.go:117] "RemoveContainer" containerID="788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"
Jul 02 14:32:23.778172 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:23.778149    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9\": container with ID starting with 788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9 not found: ID does not exist" containerID="788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"
Jul 02 14:32:23.778221 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.778181    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"} err="failed to get container status \"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9\": rpc error: code = NotFound desc = could not find container \"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9\": container with ID starting with 788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9 not found: ID does not exist"
Jul 02 14:32:23.778221 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.778202    2583 scope.go:117] "RemoveContainer" containerID="7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"
Jul 02 14:32:23.778415 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.778395    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4"} err="failed to get container status \"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4\": rpc error: code = NotFound desc = could not find container \"7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4\": container with ID starting with 7ee08e72d77002c2bf594c33e1bfcfb38b21983522c81bd6e6f82f841db6b3f4 not found: ID does not exist"
Jul 02 14:32:23.778452 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.778418    2583 scope.go:117] "RemoveContainer" containerID="788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"
Jul 02 14:32:23.778624 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.778605    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9"} err="failed to get container status \"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9\": rpc error: code = NotFound desc = could not find container \"788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9\": container with ID starting with 788661045d59f3d3bd586ed7f2e028052d4ea10d83a45b0f5e959b4f3d472db9 not found: ID does not exist"
Jul 02 14:32:23.783403 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.783382    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"]
Jul 02 14:32:23.786509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.786488    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-f89aa-predictor-7b4979f9b7-75gvg"]
Jul 02 14:32:23.833716 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.833690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:23.835989 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:23.835972    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls\") pod \"isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:24.060839 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:24.060750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:24.150157 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:24.150129    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" path="/var/lib/kubelet/pods/d24544ce-6fc5-4cd7-a1fb-1f7fc65702de/volumes"
Jul 02 14:32:24.193499 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:24.193444    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"]
Jul 02 14:32:24.195554 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:32:24.195529    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90c7b87_d358_4e52_8f9c_f66207d43db1.slice/crio-d9a1ccb3f5bd7f6bf7b4a7e2ae761870122fa151dd3c980056ef4de0645c909f WatchSource:0}: Error finding container d9a1ccb3f5bd7f6bf7b4a7e2ae761870122fa151dd3c980056ef4de0645c909f: Status 404 returned error can't find the container with id d9a1ccb3f5bd7f6bf7b4a7e2ae761870122fa151dd3c980056ef4de0645c909f
Jul 02 14:32:24.770521 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:24.770480    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerStarted","Data":"df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f"}
Jul 02 14:32:24.770668 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:24.770563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerStarted","Data":"d9a1ccb3f5bd7f6bf7b4a7e2ae761870122fa151dd3c980056ef4de0645c909f"}
Jul 02 14:32:25.445513 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:25.445469    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 14:32:27.784301 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:27.784272    2583 generic.go:356] "Generic (PLEG): container finished" podID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerID="ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0" exitCode=0
Jul 02 14:32:27.784301 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:27.784305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerDied","Data":"ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0"}
Jul 02 14:32:28.788901 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:28.788872    2583 generic.go:356] "Generic (PLEG): container finished" podID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerID="df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f" exitCode=0
Jul 02 14:32:28.789288 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:28.788925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerDied","Data":"df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f"}
Jul 02 14:32:29.794092 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:29.794057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerStarted","Data":"cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a"}
Jul 02 14:32:29.794523 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:29.794115    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerStarted","Data":"e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2"}
Jul 02 14:32:29.794523 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:29.794377    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:29.794523 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:29.794501    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:29.795709 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:29.795685    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:32:29.815987 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:29.815945    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podStartSLOduration=3.106871165 podStartE2EDuration="6.815931005s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70905984s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:32:29.813989392 +0000 UTC m=+1012.069394744" watchObservedRunningTime="2026-07-02 14:32:29.815931005 +0000 UTC m=+1012.071336346"
Jul 02 14:32:30.445185 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:30.445145    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 14:32:30.450529 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:30.450496    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:32:30.450841 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:30.450810    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:32:30.797714 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:30.797673    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:32:35.444673 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:35.444637    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 14:32:35.445081 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:35.444797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:32:35.802548 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:35.802517    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:32:35.803123 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:35.803091    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:32:40.444815 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:40.444753    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 14:32:40.450140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:40.450116    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:32:40.450527 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:40.450505    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:32:45.445265 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:45.445224    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 14:32:45.803120 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:45.803084    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:32:50.444932 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:50.444888    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 02 14:32:50.450300 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:50.450270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 02 14:32:50.450402 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:50.450388    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:32:50.450585 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:50.450569    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 14:32:50.450681 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:50.450648    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:32:53.206500 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.206478    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:32:53.358293 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358210    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kserve-provision-location\") pod \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") "
Jul 02 14:32:53.358293 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358240    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-cabundle-cert\") pod \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") "
Jul 02 14:32:53.358293 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-rbac-proxy-sar-config\") pod \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") "
Jul 02 14:32:53.358564 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358451    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-proxy-tls\") pod \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") "
Jul 02 14:32:53.358564 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358495    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkvmb\" (UniqueName: \"kubernetes.io/projected/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-api-access-vkvmb\") pod \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\" (UID: \"6fee74b3-e784-45c0-82bb-c2cfe9c68c68\") "
Jul 02 14:32:53.358664 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358584    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fee74b3-e784-45c0-82bb-c2cfe9c68c68" (UID: "6fee74b3-e784-45c0-82bb-c2cfe9c68c68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:32:53.358664 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358613    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fee74b3-e784-45c0-82bb-c2cfe9c68c68" (UID: "6fee74b3-e784-45c0-82bb-c2cfe9c68c68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:32:53.358664 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358591    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fee74b3-e784-45c0-82bb-c2cfe9c68c68" (UID: "6fee74b3-e784-45c0-82bb-c2cfe9c68c68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:32:53.358795 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358721    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:53.358795 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358739    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:53.358795 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.358754    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:53.360509 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.360486    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fee74b3-e784-45c0-82bb-c2cfe9c68c68" (UID: "6fee74b3-e784-45c0-82bb-c2cfe9c68c68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:32:53.360600 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.360580    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-api-access-vkvmb" (OuterVolumeSpecName: "kube-api-access-vkvmb") pod "6fee74b3-e784-45c0-82bb-c2cfe9c68c68" (UID: "6fee74b3-e784-45c0-82bb-c2cfe9c68c68"). InnerVolumeSpecName "kube-api-access-vkvmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:32:53.459724 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.459690    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:53.459724 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.459717    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkvmb\" (UniqueName: \"kubernetes.io/projected/6fee74b3-e784-45c0-82bb-c2cfe9c68c68-kube-api-access-vkvmb\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:32:53.879886 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.879851    2583 generic.go:356] "Generic (PLEG): container finished" podID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerID="3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba" exitCode=0
Jul 02 14:32:53.880140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.879918    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"
Jul 02 14:32:53.880140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.879938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerDied","Data":"3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba"}
Jul 02 14:32:53.880140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.880003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh" event={"ID":"6fee74b3-e784-45c0-82bb-c2cfe9c68c68","Type":"ContainerDied","Data":"66fcbe53eef256a8d3e88a0ff02147154c5a84b3fafeef1c6aa07be76f64202f"}
Jul 02 14:32:53.880140 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.880036    2583 scope.go:117] "RemoveContainer" containerID="3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba"
Jul 02 14:32:53.887944 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.887747    2583 scope.go:117] "RemoveContainer" containerID="1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767"
Jul 02 14:32:53.895008 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.894992    2583 scope.go:117] "RemoveContainer" containerID="ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0"
Jul 02 14:32:53.901937 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.901890    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"]
Jul 02 14:32:53.902690 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.902674    2583 scope.go:117] "RemoveContainer" containerID="b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a"
Jul 02 14:32:53.903850 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.903832    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-f89aa-predictor-55d69dbdfc-s8qqh"]
Jul 02 14:32:53.909188 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.909173    2583 scope.go:117] "RemoveContainer" containerID="3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba"
Jul 02 14:32:53.909441 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:53.909424    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba\": container with ID starting with 3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba not found: ID does not exist" containerID="3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba"
Jul 02 14:32:53.909487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.909451    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba"} err="failed to get container status \"3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba\": rpc error: code = NotFound desc = could not find container \"3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba\": container with ID starting with 3ba936af57e8a7f90f40f1535af6c0ebbcca5060b7b7460675c8f03aa6b828ba not found: ID does not exist"
Jul 02 14:32:53.909487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.909468    2583 scope.go:117] "RemoveContainer" containerID="1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767"
Jul 02 14:32:53.909687 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:53.909669    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767\": container with ID starting with 1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767 not found: ID does not exist" containerID="1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767"
Jul 02 14:32:53.909730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.909695    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767"} err="failed to get container status \"1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767\": rpc error: code = NotFound desc = could not find container \"1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767\": container with ID starting with 1035ad7ecd2506ccff880722e8b183b780f77ea81a140364792691a28b53b767 not found: ID does not exist"
Jul 02 14:32:53.909730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.909710    2583 scope.go:117] "RemoveContainer" containerID="ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0"
Jul 02 14:32:53.909959 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:53.909944    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0\": container with ID starting with ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0 not found: ID does not exist" containerID="ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0"
Jul 02 14:32:53.910021 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.909967    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0"} err="failed to get container status \"ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0\": rpc error: code = NotFound desc = could not find container \"ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0\": container with ID starting with ab24e3a20901760b4851986dbe821fdc4feb3ea6c7e3fecf109b25836166bff0 not found: ID does not exist"
Jul 02 14:32:53.910021 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.909986    2583 scope.go:117] "RemoveContainer" containerID="b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a"
Jul 02 14:32:53.910206 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:32:53.910190    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a\": container with ID starting with b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a not found: ID does not exist" containerID="b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a"
Jul 02 14:32:53.910244 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:53.910210    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a"} err="failed to get container status \"b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a\": rpc error: code = NotFound desc = could not find container \"b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a\": container with ID starting with b9c54c850f4364f349652ff2edd9e67ac435b9d64c7fd98eb43d8a089e13178a not found: ID does not exist"
Jul 02 14:32:54.148130 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:54.148058    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" path="/var/lib/kubelet/pods/6fee74b3-e784-45c0-82bb-c2cfe9c68c68/volumes"
Jul 02 14:32:55.804040 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:32:55.804001    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:33:05.803862 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:33:05.803824    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:33:15.804033 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:33:15.803995    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:33:25.803884 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:33:25.803842    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:33:35.803975 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:33:35.803895    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:33:45.803668 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:33:45.803622    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:33:52.143826 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:33:52.143760    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:34:02.143874 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:34:02.143826    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:34:12.144016 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:34:12.143973    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:34:22.144184 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:34:22.144139    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:34:32.144635 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:34:32.144584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:34:42.144345 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:34:42.144302    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:34:52.143881 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:34:52.143839    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:35:02.149214 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:02.149188    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:35:03.327156 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:03.327120    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"]
Jul 02 14:35:03.327606 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:03.327398    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" containerID="cri-o://e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2" gracePeriod=30
Jul 02 14:35:03.327606 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:03.327422    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kube-rbac-proxy" containerID="cri-o://cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a" gracePeriod=30
Jul 02 14:35:04.338075 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:04.338038    2583 generic.go:356] "Generic (PLEG): container finished" podID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerID="cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a" exitCode=2
Jul 02 14:35:04.338438 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:04.338109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerDied","Data":"cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a"}
Jul 02 14:35:05.797960 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:05.797921    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 02 14:35:08.440757 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.440721    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"]
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441013    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441026    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441042    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kube-rbac-proxy"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441047    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kube-rbac-proxy"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441056    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kserve-container"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441062    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kserve-container"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441070    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441075    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441083    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441088    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441095    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="storage-initializer"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441100    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="storage-initializer"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441142    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kube-rbac-proxy"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441150    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="agent"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441158    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24544ce-6fc5-4cd7-a1fb-1f7fc65702de" containerName="kserve-container"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441164    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kserve-container"
Jul 02 14:35:08.441237 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.441169    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fee74b3-e784-45c0-82bb-c2cfe9c68c68" containerName="kube-rbac-proxy"
Jul 02 14:35:08.444001 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.443984    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.446556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.446521    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-571456-predictor-serving-cert\""
Jul 02 14:35:08.446684 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.446616    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-571456-kube-rbac-proxy-sar-config\""
Jul 02 14:35:08.454394 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.454372    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"]
Jul 02 14:35:08.527014 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.526981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhq5\" (UniqueName: \"kubernetes.io/projected/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-api-access-cwhq5\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.527180 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.527040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-proxy-tls\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.527180 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.527091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kserve-provision-location\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.527180 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.527116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-cabundle-cert\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.527180 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.527143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.627975 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.627943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhq5\" (UniqueName: \"kubernetes.io/projected/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-api-access-cwhq5\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.628160 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.627997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-proxy-tls\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.628160 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.628025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kserve-provision-location\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.628160 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.628042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-cabundle-cert\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.628160 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.628061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.628441 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.628420    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kserve-provision-location\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.628808 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.628761    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-rbac-proxy-sar-config\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.628903 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.628795    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-cabundle-cert\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.630534 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.630516    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-proxy-tls\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.637405 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.637384    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhq5\" (UniqueName: \"kubernetes.io/projected/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-api-access-cwhq5\") pod \"isvc-primary-571456-predictor-bdc8f5d66-rdw9f\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") " pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.755652 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.755617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:08.884793 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:08.884745    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"]
Jul 02 14:35:08.887699 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:35:08.887673    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b8e2c5_d5ef_4f95_8130_fe918dade8b6.slice/crio-d87618db22f48898b936edfb9923f58e7615f7082d977d606a739d58fe12b87b WatchSource:0}: Error finding container d87618db22f48898b936edfb9923f58e7615f7082d977d606a739d58fe12b87b: Status 404 returned error can't find the container with id d87618db22f48898b936edfb9923f58e7615f7082d977d606a739d58fe12b87b
Jul 02 14:35:09.359414 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:09.359379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerStarted","Data":"4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609"}
Jul 02 14:35:09.359582 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:09.359450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerStarted","Data":"d87618db22f48898b936edfb9923f58e7615f7082d977d606a739d58fe12b87b"}
Jul 02 14:35:10.798560 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:10.798522    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 02 14:35:12.144072 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:12.144033    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 02 14:35:12.967737 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:12.967710    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:35:13.059621 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.059590    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlr89\" (UniqueName: \"kubernetes.io/projected/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-api-access-nlr89\") pod \"f90c7b87-d358-4e52-8f9c-f66207d43db1\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") "
Jul 02 14:35:13.059940 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.059645    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90c7b87-d358-4e52-8f9c-f66207d43db1-kserve-provision-location\") pod \"f90c7b87-d358-4e52-8f9c-f66207d43db1\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") "
Jul 02 14:35:13.059940 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.059672    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-rbac-proxy-sar-config\") pod \"f90c7b87-d358-4e52-8f9c-f66207d43db1\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") "
Jul 02 14:35:13.059940 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.059726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls\") pod \"f90c7b87-d358-4e52-8f9c-f66207d43db1\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") "
Jul 02 14:35:13.059940 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.059747    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-cabundle-cert\") pod \"f90c7b87-d358-4e52-8f9c-f66207d43db1\" (UID: \"f90c7b87-d358-4e52-8f9c-f66207d43db1\") "
Jul 02 14:35:13.060149 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.060086    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90c7b87-d358-4e52-8f9c-f66207d43db1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f90c7b87-d358-4e52-8f9c-f66207d43db1" (UID: "f90c7b87-d358-4e52-8f9c-f66207d43db1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:35:13.060149 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.060119    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f90c7b87-d358-4e52-8f9c-f66207d43db1" (UID: "f90c7b87-d358-4e52-8f9c-f66207d43db1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:35:13.060217 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.060176    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f90c7b87-d358-4e52-8f9c-f66207d43db1" (UID: "f90c7b87-d358-4e52-8f9c-f66207d43db1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:35:13.061833 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.061811    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-api-access-nlr89" (OuterVolumeSpecName: "kube-api-access-nlr89") pod "f90c7b87-d358-4e52-8f9c-f66207d43db1" (UID: "f90c7b87-d358-4e52-8f9c-f66207d43db1"). InnerVolumeSpecName "kube-api-access-nlr89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:35:13.061911 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.061844    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f90c7b87-d358-4e52-8f9c-f66207d43db1" (UID: "f90c7b87-d358-4e52-8f9c-f66207d43db1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:35:13.160513 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.160426    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90c7b87-d358-4e52-8f9c-f66207d43db1-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:35:13.160513 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.160460    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:35:13.160513 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.160470    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlr89\" (UniqueName: \"kubernetes.io/projected/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-api-access-nlr89\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:35:13.160513 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.160479    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90c7b87-d358-4e52-8f9c-f66207d43db1-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:35:13.160513 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.160488    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90c7b87-d358-4e52-8f9c-f66207d43db1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:35:13.376007 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.375971    2583 generic.go:356] "Generic (PLEG): container finished" podID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerID="e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2" exitCode=0
Jul 02 14:35:13.376176 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.376043    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"
Jul 02 14:35:13.376176 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.376054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerDied","Data":"e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2"}
Jul 02 14:35:13.376176 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.376118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6" event={"ID":"f90c7b87-d358-4e52-8f9c-f66207d43db1","Type":"ContainerDied","Data":"d9a1ccb3f5bd7f6bf7b4a7e2ae761870122fa151dd3c980056ef4de0645c909f"}
Jul 02 14:35:13.376176 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.376138    2583 scope.go:117] "RemoveContainer" containerID="cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a"
Jul 02 14:35:13.377591 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.377563    2583 generic.go:356] "Generic (PLEG): container finished" podID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerID="4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609" exitCode=0
Jul 02 14:35:13.377700 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.377642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerDied","Data":"4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609"}
Jul 02 14:35:13.385474 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.385456    2583 scope.go:117] "RemoveContainer" containerID="e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2"
Jul 02 14:35:13.394647 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.394628    2583 scope.go:117] "RemoveContainer" containerID="df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f"
Jul 02 14:35:13.403612 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.403595    2583 scope.go:117] "RemoveContainer" containerID="cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a"
Jul 02 14:35:13.403898 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:35:13.403880    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a\": container with ID starting with cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a not found: ID does not exist" containerID="cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a"
Jul 02 14:35:13.403981 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.403911    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a"} err="failed to get container status \"cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a\": rpc error: code = NotFound desc = could not find container \"cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a\": container with ID starting with cd7ae3af61d50155462317f402937a08f662baffd26a45d20830c273d6ffd13a not found: ID does not exist"
Jul 02 14:35:13.403981 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.403934    2583 scope.go:117] "RemoveContainer" containerID="e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2"
Jul 02 14:35:13.404175 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:35:13.404158    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2\": container with ID starting with e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2 not found: ID does not exist" containerID="e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2"
Jul 02 14:35:13.404216 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.404183    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2"} err="failed to get container status \"e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2\": rpc error: code = NotFound desc = could not find container \"e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2\": container with ID starting with e52cc232176549d056debde0dae14c6b7d69411f8074d48dcd55595e8d8ebab2 not found: ID does not exist"
Jul 02 14:35:13.404216 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.404197    2583 scope.go:117] "RemoveContainer" containerID="df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f"
Jul 02 14:35:13.404442 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:35:13.404426    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f\": container with ID starting with df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f not found: ID does not exist" containerID="df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f"
Jul 02 14:35:13.404494 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.404445    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f"} err="failed to get container status \"df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f\": rpc error: code = NotFound desc = could not find container \"df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f\": container with ID starting with df0a035e34c0e49adca005328c297e4dcabe4c8cf5f4872af592bd981bdf132f not found: ID does not exist"
Jul 02 14:35:13.411604 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.411407    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"]
Jul 02 14:35:13.413186 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:13.413167    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-a9dfc-predictor-7c997cd95c-2hmp6"]
Jul 02 14:35:14.149209 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:14.149179    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" path="/var/lib/kubelet/pods/f90c7b87-d358-4e52-8f9c-f66207d43db1/volumes"
Jul 02 14:35:14.384224 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:14.384191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerStarted","Data":"065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712"}
Jul 02 14:35:14.384592 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:14.384256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerStarted","Data":"4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1"}
Jul 02 14:35:14.384592 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:14.384436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:14.406580 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:14.406492    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podStartSLOduration=2.814751089 podStartE2EDuration="6.406475719s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.59172463s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:35:14.404862844 +0000 UTC m=+1176.660268189" watchObservedRunningTime="2026-07-02 14:35:14.406475719 +0000 UTC m=+1176.661881059"
Jul 02 14:35:15.387358 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:15.387324    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:15.388624 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:15.388598    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:35:16.390364 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:16.390326    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:35:21.394471 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:21.394436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:35:21.394960 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:21.394932    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:35:31.395188 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:31.395137    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:35:38.152989 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:38.152962    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:35:38.153986 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:38.153967    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:35:41.395563 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:41.395526    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:35:51.394848 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:35:51.394811    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:36:01.395733 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:01.395692    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:36:11.395036 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:11.395002    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:36:21.395890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:21.395848    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:36:31.396018 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:31.395934    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:36:38.662784 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.662731    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"]
Jul 02 14:36:38.663159 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663133    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="storage-initializer"
Jul 02 14:36:38.663159 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663149    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="storage-initializer"
Jul 02 14:36:38.663236 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663164    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kube-rbac-proxy"
Jul 02 14:36:38.663236 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663173    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kube-rbac-proxy"
Jul 02 14:36:38.663236 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663205    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container"
Jul 02 14:36:38.663236 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663214    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container"
Jul 02 14:36:38.663355 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663271    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kserve-container"
Jul 02 14:36:38.663355 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.663283    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90c7b87-d358-4e52-8f9c-f66207d43db1" containerName="kube-rbac-proxy"
Jul 02 14:36:38.666543 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.666526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.669708 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.669688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-571456\""
Jul 02 14:36:38.669708 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.669700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-571456-dockercfg-xc7vf\""
Jul 02 14:36:38.669708 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.669704    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-571456-predictor-serving-cert\""
Jul 02 14:36:38.670028 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.669848    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-571456-kube-rbac-proxy-sar-config\""
Jul 02 14:36:38.670255 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.670236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.670302 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.670277    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.670353 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.670309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-cabundle-cert\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.670431 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.670413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8gv\" (UniqueName: \"kubernetes.io/projected/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-api-access-gz8gv\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.670487 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.670446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kserve-provision-location\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.678645 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.678621    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"]
Jul 02 14:36:38.770924 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.770889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.771127 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.770933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.771127 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.770962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-cabundle-cert\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.771127 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.771002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8gv\" (UniqueName: \"kubernetes.io/projected/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-api-access-gz8gv\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.771127 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.771024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kserve-provision-location\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.771127 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:36:38.771105    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-571456-predictor-serving-cert: secret "isvc-secondary-571456-predictor-serving-cert" not found
Jul 02 14:36:38.771428 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:36:38.771187    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls podName:22c7c681-ac3c-4b3e-90ac-ef27d37fe65e nodeName:}" failed. No retries permitted until 2026-07-02 14:36:39.271163273 +0000 UTC m=+1261.526568612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls") pod "isvc-secondary-571456-predictor-7b4f79889c-pdzpt" (UID: "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e") : secret "isvc-secondary-571456-predictor-serving-cert" not found
Jul 02 14:36:38.771498 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.771468    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kserve-provision-location\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.771637 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.771620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-cabundle-cert\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.771673 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.771617    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:38.781046 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:38.781022    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8gv\" (UniqueName: \"kubernetes.io/projected/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-api-access-gz8gv\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:39.279310 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:39.279270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:39.281641 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:39.281620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls\") pod \"isvc-secondary-571456-predictor-7b4f79889c-pdzpt\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") " pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:39.577196 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:39.577108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:39.707981 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:39.707958    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"]
Jul 02 14:36:39.712454 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:36:39.712427    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c7c681_ac3c_4b3e_90ac_ef27d37fe65e.slice/crio-c31abefec39e24203d32e0857aa894bd8688a20d8d2f67eaf3d490a36b5d4e5f WatchSource:0}: Error finding container c31abefec39e24203d32e0857aa894bd8688a20d8d2f67eaf3d490a36b5d4e5f: Status 404 returned error can't find the container with id c31abefec39e24203d32e0857aa894bd8688a20d8d2f67eaf3d490a36b5d4e5f
Jul 02 14:36:40.684721 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:40.684686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt" event={"ID":"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e","Type":"ContainerStarted","Data":"6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87"}
Jul 02 14:36:40.685090 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:40.684748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt" event={"ID":"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e","Type":"ContainerStarted","Data":"c31abefec39e24203d32e0857aa894bd8688a20d8d2f67eaf3d490a36b5d4e5f"}
Jul 02 14:36:46.707162 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:46.707134    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/storage-initializer/0.log"
Jul 02 14:36:46.707525 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:46.707170    2583 generic.go:356] "Generic (PLEG): container finished" podID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerID="6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87" exitCode=1
Jul 02 14:36:46.707525 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:46.707196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt" event={"ID":"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e","Type":"ContainerDied","Data":"6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87"}
Jul 02 14:36:47.713320 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:47.713295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/storage-initializer/0.log"
Jul 02 14:36:47.713682 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:47.713377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt" event={"ID":"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e","Type":"ContainerStarted","Data":"80aa75db1264356a05b395102a81b1593706f2c51c15b5ca903bbdce20d42fe0"}
Jul 02 14:36:50.725108 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:50.725080    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/storage-initializer/1.log"
Jul 02 14:36:50.725510 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:50.725495    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/storage-initializer/0.log"
Jul 02 14:36:50.725570 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:50.725527    2583 generic.go:356] "Generic (PLEG): container finished" podID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerID="80aa75db1264356a05b395102a81b1593706f2c51c15b5ca903bbdce20d42fe0" exitCode=1
Jul 02 14:36:50.725617 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:50.725599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt" event={"ID":"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e","Type":"ContainerDied","Data":"80aa75db1264356a05b395102a81b1593706f2c51c15b5ca903bbdce20d42fe0"}
Jul 02 14:36:50.725669 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:50.725659    2583 scope.go:117] "RemoveContainer" containerID="6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87"
Jul 02 14:36:50.726022 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:50.726005    2583 scope.go:117] "RemoveContainer" containerID="6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87"
Jul 02 14:36:50.735818 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:36:50.735795    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-571456-predictor-7b4f79889c-pdzpt_kserve-ci-e2e-test(22c7c681-ac3c-4b3e-90ac-ef27d37fe65e)\"" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e"
Jul 02 14:36:50.736004 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:36:50.735981    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_kserve-ci-e2e-test_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e_0 in pod sandbox c31abefec39e24203d32e0857aa894bd8688a20d8d2f67eaf3d490a36b5d4e5f from index: no such id: '6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87'" containerID="6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87"
Jul 02 14:36:50.736058 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:50.736019    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_kserve-ci-e2e-test_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e_0 in pod sandbox c31abefec39e24203d32e0857aa894bd8688a20d8d2f67eaf3d490a36b5d4e5f from index: no such id: '6624624168d4a7f0f81f1d1757cf6db0fe404154eabc89a460572bc16b703e87'"
Jul 02 14:36:51.730864 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:51.730836    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/storage-initializer/1.log"
Jul 02 14:36:58.689264 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.689225    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"]
Jul 02 14:36:58.744519 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.744483    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"]
Jul 02 14:36:58.745982 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.744793    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" containerID="cri-o://4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1" gracePeriod=30
Jul 02 14:36:58.745982 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.744861    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kube-rbac-proxy" containerID="cri-o://065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712" gracePeriod=30
Jul 02 14:36:58.863289 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.863272    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/storage-initializer/1.log"
Jul 02 14:36:58.865396 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.865380    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:58.918119 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918088    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-cabundle-cert\") pod \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") "
Jul 02 14:36:58.918280 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918144    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kserve-provision-location\") pod \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") "
Jul 02 14:36:58.918280 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918178    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz8gv\" (UniqueName: \"kubernetes.io/projected/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-api-access-gz8gv\") pod \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") "
Jul 02 14:36:58.918280 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls\") pod \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") "
Jul 02 14:36:58.918456 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918364    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-rbac-proxy-sar-config\") pod \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\" (UID: \"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e\") "
Jul 02 14:36:58.918456 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918370    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" (UID: "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:36:58.918557 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918480    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" (UID: "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:36:58.918671 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918651    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:36:58.918671 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918670    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:36:58.918746 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.918680    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" (UID: "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:36:58.920301 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.920277    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-api-access-gz8gv" (OuterVolumeSpecName: "kube-api-access-gz8gv") pod "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" (UID: "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e"). InnerVolumeSpecName "kube-api-access-gz8gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:36:58.920459 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:58.920440    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" (UID: "22c7c681-ac3c-4b3e-90ac-ef27d37fe65e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:36:59.019547 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.019514    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz8gv\" (UniqueName: \"kubernetes.io/projected/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-api-access-gz8gv\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:36:59.019547 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.019545    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:36:59.019547 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.019555    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:36:59.759588 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.759563    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-571456-predictor-7b4f79889c-pdzpt_22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/storage-initializer/1.log"
Jul 02 14:36:59.760067 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.759652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt" event={"ID":"22c7c681-ac3c-4b3e-90ac-ef27d37fe65e","Type":"ContainerDied","Data":"c31abefec39e24203d32e0857aa894bd8688a20d8d2f67eaf3d490a36b5d4e5f"}
Jul 02 14:36:59.760067 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.759663    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"
Jul 02 14:36:59.760067 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.759705    2583 scope.go:117] "RemoveContainer" containerID="80aa75db1264356a05b395102a81b1593706f2c51c15b5ca903bbdce20d42fe0"
Jul 02 14:36:59.761889 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.761864    2583 generic.go:356] "Generic (PLEG): container finished" podID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerID="065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712" exitCode=2
Jul 02 14:36:59.762029 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.761908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerDied","Data":"065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712"}
Jul 02 14:36:59.798637 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.798607    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"]
Jul 02 14:36:59.803943 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:36:59.803916    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-571456-predictor-7b4f79889c-pdzpt"]
Jul 02 14:37:00.148834 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:00.148742    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" path="/var/lib/kubelet/pods/22c7c681-ac3c-4b3e-90ac-ef27d37fe65e/volumes"
Jul 02 14:37:01.391213 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:01.391176    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 02 14:37:01.395568 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:01.395545    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 02 14:37:03.384934 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.384911    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:37:03.451256 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451173    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-rbac-proxy-sar-config\") pod \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") "
Jul 02 14:37:03.451256 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451212    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-cabundle-cert\") pod \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") "
Jul 02 14:37:03.451256 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451237    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwhq5\" (UniqueName: \"kubernetes.io/projected/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-api-access-cwhq5\") pod \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") "
Jul 02 14:37:03.451533 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451429    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-proxy-tls\") pod \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") "
Jul 02 14:37:03.451533 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451458    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kserve-provision-location\") pod \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\" (UID: \"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6\") "
Jul 02 14:37:03.451642 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451563    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" (UID: "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:03.451691 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451635    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" (UID: "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:03.451691 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451682    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:03.451876 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.451853    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" (UID: "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:37:03.453338 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.453313    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-api-access-cwhq5" (OuterVolumeSpecName: "kube-api-access-cwhq5") pod "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" (UID: "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6"). InnerVolumeSpecName "kube-api-access-cwhq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:37:03.453338 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.453321    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" (UID: "b4b8e2c5-d5ef-4f95-8130-fe918dade8b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:37:03.552532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.552500    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:03.552532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.552524    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhq5\" (UniqueName: \"kubernetes.io/projected/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kube-api-access-cwhq5\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:03.552532 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.552534    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:03.552756 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.552546    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:03.781172 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.781130    2583 generic.go:356] "Generic (PLEG): container finished" podID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerID="4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1" exitCode=0
Jul 02 14:37:03.781357 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.781208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerDied","Data":"4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1"}
Jul 02 14:37:03.781357 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.781226    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"
Jul 02 14:37:03.781357 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.781291    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f" event={"ID":"b4b8e2c5-d5ef-4f95-8130-fe918dade8b6","Type":"ContainerDied","Data":"d87618db22f48898b936edfb9923f58e7615f7082d977d606a739d58fe12b87b"}
Jul 02 14:37:03.781357 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.781319    2583 scope.go:117] "RemoveContainer" containerID="065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712"
Jul 02 14:37:03.790823 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.790714    2583 scope.go:117] "RemoveContainer" containerID="4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1"
Jul 02 14:37:03.798981 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.798964    2583 scope.go:117] "RemoveContainer" containerID="4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609"
Jul 02 14:37:03.804267 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.804241    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"]
Jul 02 14:37:03.806934 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.806914    2583 scope.go:117] "RemoveContainer" containerID="065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712"
Jul 02 14:37:03.807228 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:03.807203    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712\": container with ID starting with 065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712 not found: ID does not exist" containerID="065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712"
Jul 02 14:37:03.807319 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.807235    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712"} err="failed to get container status \"065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712\": rpc error: code = NotFound desc = could not find container \"065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712\": container with ID starting with 065eab33ff933508bfce42f05d1f4ad0a1271dfcaaccec48764e8eb2e2c9b712 not found: ID does not exist"
Jul 02 14:37:03.807319 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.807253    2583 scope.go:117] "RemoveContainer" containerID="4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1"
Jul 02 14:37:03.807512 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:03.807494    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1\": container with ID starting with 4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1 not found: ID does not exist" containerID="4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1"
Jul 02 14:37:03.807584 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.807516    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1"} err="failed to get container status \"4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1\": rpc error: code = NotFound desc = could not find container \"4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1\": container with ID starting with 4ca510336bad8509707390a409c411881d1f3f598bb6b430401f2a284b4f84a1 not found: ID does not exist"
Jul 02 14:37:03.807584 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.807534    2583 scope.go:117] "RemoveContainer" containerID="4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609"
Jul 02 14:37:03.807842 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:03.807814    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609\": container with ID starting with 4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609 not found: ID does not exist" containerID="4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609"
Jul 02 14:37:03.807920 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.807850    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609"} err="failed to get container status \"4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609\": rpc error: code = NotFound desc = could not find container \"4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609\": container with ID starting with 4e6d9ee7fba0a073e020212aacb7735474b60d9a24a2a8407d4f944faa3f3609 not found: ID does not exist"
Jul 02 14:37:03.813167 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.813143    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-571456-predictor-bdc8f5d66-rdw9f"]
Jul 02 14:37:03.885178 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885146    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"]
Jul 02 14:37:03.885420 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885407    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerName="storage-initializer"
Jul 02 14:37:03.885465 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885422    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerName="storage-initializer"
Jul 02 14:37:03.885465 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885438    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kube-rbac-proxy"
Jul 02 14:37:03.885465 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885445    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kube-rbac-proxy"
Jul 02 14:37:03.885465 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885459    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="storage-initializer"
Jul 02 14:37:03.885465 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885465    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="storage-initializer"
Jul 02 14:37:03.885609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885475    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container"
Jul 02 14:37:03.885609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885480    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container"
Jul 02 14:37:03.885609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885522    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerName="storage-initializer"
Jul 02 14:37:03.885609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885532    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kserve-container"
Jul 02 14:37:03.885609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885541    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerName="storage-initializer"
Jul 02 14:37:03.885609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885548    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" containerName="kube-rbac-proxy"
Jul 02 14:37:03.885609 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885606    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerName="storage-initializer"
Jul 02 14:37:03.885974 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.885615    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c7c681-ac3c-4b3e-90ac-ef27d37fe65e" containerName="storage-initializer"
Jul 02 14:37:03.890408 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.890389    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:03.893546 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.893479    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-0ecf21\""
Jul 02 14:37:03.893546 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.893499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:37:03.893546 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.893503    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-0ecf21-kube-rbac-proxy-sar-config\""
Jul 02 14:37:03.893810 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.893508    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:37:03.893810 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.893511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-0ecf21-dockercfg-sbgn4\""
Jul 02 14:37:03.893909 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.893840    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-0ecf21-predictor-serving-cert\""
Jul 02 14:37:03.893909 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.893881    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:37:03.899001 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.898938    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"]
Jul 02 14:37:03.956123 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.956092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933efd14-d42d-467d-a7d1-3ae478991c3a-kserve-provision-location\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:03.956309 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.956131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp7dj\" (UniqueName: \"kubernetes.io/projected/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-api-access-kp7dj\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:03.956309 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.956192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:03.956309 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.956240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-cabundle-cert\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:03.956309 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:03.956263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.057461 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.057385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933efd14-d42d-467d-a7d1-3ae478991c3a-kserve-provision-location\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.057461 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.057424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp7dj\" (UniqueName: \"kubernetes.io/projected/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-api-access-kp7dj\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.057639 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.057599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.057639 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.057626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-cabundle-cert\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.057709 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.057647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.057809 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.057760    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933efd14-d42d-467d-a7d1-3ae478991c3a-kserve-provision-location\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.058246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.058225    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.058319 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.058232    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-cabundle-cert\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.060229 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.060206    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.065827 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.065810    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp7dj\" (UniqueName: \"kubernetes.io/projected/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-api-access-kp7dj\") pod \"isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") " pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.147754 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.147719    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b8e2c5-d5ef-4f95-8130-fe918dade8b6" path="/var/lib/kubelet/pods/b4b8e2c5-d5ef-4f95-8130-fe918dade8b6/volumes"
Jul 02 14:37:04.202124 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.202094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:04.327950 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.327927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"]
Jul 02 14:37:04.329925 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:37:04.329898    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933efd14_d42d_467d_a7d1_3ae478991c3a.slice/crio-b399454b6e446481533430483dbb715de6e2570800537f50972b78134c58b9f3 WatchSource:0}: Error finding container b399454b6e446481533430483dbb715de6e2570800537f50972b78134c58b9f3: Status 404 returned error can't find the container with id b399454b6e446481533430483dbb715de6e2570800537f50972b78134c58b9f3
Jul 02 14:37:04.785853 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.785822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" event={"ID":"933efd14-d42d-467d-a7d1-3ae478991c3a","Type":"ContainerStarted","Data":"133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41"}
Jul 02 14:37:04.786322 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:04.785883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" event={"ID":"933efd14-d42d-467d-a7d1-3ae478991c3a","Type":"ContainerStarted","Data":"b399454b6e446481533430483dbb715de6e2570800537f50972b78134c58b9f3"}
Jul 02 14:37:07.798110 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:07.798083    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp_933efd14-d42d-467d-a7d1-3ae478991c3a/storage-initializer/0.log"
Jul 02 14:37:07.798479 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:07.798124    2583 generic.go:356] "Generic (PLEG): container finished" podID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerID="133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41" exitCode=1
Jul 02 14:37:07.798479 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:07.798180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" event={"ID":"933efd14-d42d-467d-a7d1-3ae478991c3a","Type":"ContainerDied","Data":"133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41"}
Jul 02 14:37:08.803955 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:08.803925    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp_933efd14-d42d-467d-a7d1-3ae478991c3a/storage-initializer/0.log"
Jul 02 14:37:08.804326 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:08.803994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" event={"ID":"933efd14-d42d-467d-a7d1-3ae478991c3a","Type":"ContainerStarted","Data":"f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700"}
Jul 02 14:37:08.893261 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:08.893219    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-serving-cert: secret "isvc-init-fail-0ecf21-predictor-serving-cert" not found
Jul 02 14:37:08.893452 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:08.893327    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls podName:933efd14-d42d-467d-a7d1-3ae478991c3a nodeName:}" failed. No retries permitted until 2026-07-02 14:37:09.39330411 +0000 UTC m=+1291.648709436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls") pod "isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a") : secret "isvc-init-fail-0ecf21-predictor-serving-cert" not found
Jul 02 14:37:08.903126 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:08.903101    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"]
Jul 02 14:37:09.396785 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:09.396742    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-serving-cert: secret "isvc-init-fail-0ecf21-predictor-serving-cert" not found
Jul 02 14:37:09.396941 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:09.396826    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls podName:933efd14-d42d-467d-a7d1-3ae478991c3a nodeName:}" failed. No retries permitted until 2026-07-02 14:37:10.396813371 +0000 UTC m=+1292.652218691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls") pod "isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a") : secret "isvc-init-fail-0ecf21-predictor-serving-cert" not found
Jul 02 14:37:09.807250 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:09.807213    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerName="storage-initializer" containerID="cri-o://f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700" gracePeriod=30
Jul 02 14:37:10.405166 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:10.405133    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-serving-cert: secret "isvc-init-fail-0ecf21-predictor-serving-cert" not found
Jul 02 14:37:10.405336 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:10.405206    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls podName:933efd14-d42d-467d-a7d1-3ae478991c3a nodeName:}" failed. No retries permitted until 2026-07-02 14:37:12.405192887 +0000 UTC m=+1294.660598207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls") pod "isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a") : secret "isvc-init-fail-0ecf21-predictor-serving-cert" not found
Jul 02 14:37:12.040896 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.040875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp_933efd14-d42d-467d-a7d1-3ae478991c3a/storage-initializer/1.log"
Jul 02 14:37:12.041299 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.041281    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp_933efd14-d42d-467d-a7d1-3ae478991c3a/storage-initializer/0.log"
Jul 02 14:37:12.043559 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.043540    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:12.117826 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.117790    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-cabundle-cert\") pod \"933efd14-d42d-467d-a7d1-3ae478991c3a\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") "
Jul 02 14:37:12.117970 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.117846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls\") pod \"933efd14-d42d-467d-a7d1-3ae478991c3a\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") "
Jul 02 14:37:12.117970 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.117877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933efd14-d42d-467d-a7d1-3ae478991c3a-kserve-provision-location\") pod \"933efd14-d42d-467d-a7d1-3ae478991c3a\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") "
Jul 02 14:37:12.118100 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.117975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-rbac-proxy-sar-config\") pod \"933efd14-d42d-467d-a7d1-3ae478991c3a\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") "
Jul 02 14:37:12.118100 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.118029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp7dj\" (UniqueName: \"kubernetes.io/projected/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-api-access-kp7dj\") pod \"933efd14-d42d-467d-a7d1-3ae478991c3a\" (UID: \"933efd14-d42d-467d-a7d1-3ae478991c3a\") "
Jul 02 14:37:12.118201 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.118133    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/933efd14-d42d-467d-a7d1-3ae478991c3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "933efd14-d42d-467d-a7d1-3ae478991c3a" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:37:12.118261 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.118246    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/933efd14-d42d-467d-a7d1-3ae478991c3a-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:12.118320 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.118250    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "933efd14-d42d-467d-a7d1-3ae478991c3a" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:12.118386 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.118362    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "933efd14-d42d-467d-a7d1-3ae478991c3a" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:37:12.119919 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.119898    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "933efd14-d42d-467d-a7d1-3ae478991c3a" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:37:12.120011 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.119994    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-api-access-kp7dj" (OuterVolumeSpecName: "kube-api-access-kp7dj") pod "933efd14-d42d-467d-a7d1-3ae478991c3a" (UID: "933efd14-d42d-467d-a7d1-3ae478991c3a"). InnerVolumeSpecName "kube-api-access-kp7dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:37:12.218684 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.218654    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:12.218684 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.218681    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp7dj\" (UniqueName: \"kubernetes.io/projected/933efd14-d42d-467d-a7d1-3ae478991c3a-kube-api-access-kp7dj\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:12.218829 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.218691    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/933efd14-d42d-467d-a7d1-3ae478991c3a-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:12.218829 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.218700    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/933efd14-d42d-467d-a7d1-3ae478991c3a-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:37:12.819120 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.819093    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp_933efd14-d42d-467d-a7d1-3ae478991c3a/storage-initializer/1.log"
Jul 02 14:37:12.819483 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.819468    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp_933efd14-d42d-467d-a7d1-3ae478991c3a/storage-initializer/0.log"
Jul 02 14:37:12.819541 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.819502    2583 generic.go:356] "Generic (PLEG): container finished" podID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerID="f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700" exitCode=1
Jul 02 14:37:12.819575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.819535    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" event={"ID":"933efd14-d42d-467d-a7d1-3ae478991c3a","Type":"ContainerDied","Data":"f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700"}
Jul 02 14:37:12.819606 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.819571    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"
Jul 02 14:37:12.819638 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.819607    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp" event={"ID":"933efd14-d42d-467d-a7d1-3ae478991c3a","Type":"ContainerDied","Data":"b399454b6e446481533430483dbb715de6e2570800537f50972b78134c58b9f3"}
Jul 02 14:37:12.819671 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.819644    2583 scope.go:117] "RemoveContainer" containerID="f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700"
Jul 02 14:37:12.827097 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.827074    2583 scope.go:117] "RemoveContainer" containerID="133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41"
Jul 02 14:37:12.834095 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.834078    2583 scope.go:117] "RemoveContainer" containerID="f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700"
Jul 02 14:37:12.834333 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:12.834318    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700\": container with ID starting with f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700 not found: ID does not exist" containerID="f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700"
Jul 02 14:37:12.834376 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.834341    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700"} err="failed to get container status \"f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700\": rpc error: code = NotFound desc = could not find container \"f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700\": container with ID starting with f2a54250a8ce914ae5a9274ccd5bc19973a4520622a01ef7701198237a063700 not found: ID does not exist"
Jul 02 14:37:12.834376 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.834357    2583 scope.go:117] "RemoveContainer" containerID="133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41"
Jul 02 14:37:12.834564 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:37:12.834549    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41\": container with ID starting with 133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41 not found: ID does not exist" containerID="133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41"
Jul 02 14:37:12.834608 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.834570    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41"} err="failed to get container status \"133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41\": rpc error: code = NotFound desc = could not find container \"133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41\": container with ID starting with 133564ce3affe10aa26382581d8d5abf7e9226ccccdc8ff4c01e6dbff25deb41 not found: ID does not exist"
Jul 02 14:37:12.855184 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.855158    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"]
Jul 02 14:37:12.862144 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:12.862119    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-0ecf21-predictor-777dcd9689-sphbp"]
Jul 02 14:37:14.094890 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.094862    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"]
Jul 02 14:37:14.095246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.095126    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerName="storage-initializer"
Jul 02 14:37:14.095246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.095137    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerName="storage-initializer"
Jul 02 14:37:14.095246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.095149    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerName="storage-initializer"
Jul 02 14:37:14.095246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.095154    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerName="storage-initializer"
Jul 02 14:37:14.095246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.095201    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerName="storage-initializer"
Jul 02 14:37:14.095408 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.095275    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" containerName="storage-initializer"
Jul 02 14:37:14.099463 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.099443    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.105504 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.105481    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-5fcfc-predictor-serving-cert\""
Jul 02 14:37:14.105702 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.105550    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:37:14.105702 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.105582    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:37:14.105876 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.105709    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6qw4\""
Jul 02 14:37:14.105876 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.105714    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:37:14.105876 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.105813    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-5fcfc-kube-rbac-proxy-sar-config\""
Jul 02 14:37:14.106786 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.106564    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:37:14.111575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.111553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"]
Jul 02 14:37:14.149134 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.149099    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="933efd14-d42d-467d-a7d1-3ae478991c3a" path="/var/lib/kubelet/pods/933efd14-d42d-467d-a7d1-3ae478991c3a/volumes"
Jul 02 14:37:14.233010 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.232977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74a4a633-302b-4a6b-aaff-b061ec2516a6-proxy-tls\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.233010 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.233013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a4a633-302b-4a6b-aaff-b061ec2516a6-kserve-provision-location\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.233201 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.233031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-cabundle-cert\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.233201 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.233108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjvg\" (UniqueName: \"kubernetes.io/projected/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-api-access-qvjvg\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.233201 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.233168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.333533 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.333494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjvg\" (UniqueName: \"kubernetes.io/projected/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-api-access-qvjvg\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.333730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.333554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.333730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.333586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74a4a633-302b-4a6b-aaff-b061ec2516a6-proxy-tls\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.333730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.333615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a4a633-302b-4a6b-aaff-b061ec2516a6-kserve-provision-location\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.333730 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.333640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-cabundle-cert\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.334088 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.334064    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a4a633-302b-4a6b-aaff-b061ec2516a6-kserve-provision-location\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.334400 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.334379    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.334446 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.334431    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-cabundle-cert\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.336088 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.336067    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74a4a633-302b-4a6b-aaff-b061ec2516a6-proxy-tls\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.342696 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.342672    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjvg\" (UniqueName: \"kubernetes.io/projected/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-api-access-qvjvg\") pod \"raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") " pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.410360 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.410261    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:14.542391 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.542273    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"]
Jul 02 14:37:14.544834 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:37:14.544806    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a4a633_302b_4a6b_aaff_b061ec2516a6.slice/crio-078891098c8364eaa67059557ead4e142eedd01370189689ea6cdb24c14d649e WatchSource:0}: Error finding container 078891098c8364eaa67059557ead4e142eedd01370189689ea6cdb24c14d649e: Status 404 returned error can't find the container with id 078891098c8364eaa67059557ead4e142eedd01370189689ea6cdb24c14d649e
Jul 02 14:37:14.830070 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.830035    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerStarted","Data":"b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d"}
Jul 02 14:37:14.830243 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:14.830091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerStarted","Data":"078891098c8364eaa67059557ead4e142eedd01370189689ea6cdb24c14d649e"}
Jul 02 14:37:18.845853 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:18.845822    2583 generic.go:356] "Generic (PLEG): container finished" podID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerID="b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d" exitCode=0
Jul 02 14:37:18.846258 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:18.845885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerDied","Data":"b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d"}
Jul 02 14:37:19.851327 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:19.851294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerStarted","Data":"4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6"}
Jul 02 14:37:19.851705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:19.851355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerStarted","Data":"6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995"}
Jul 02 14:37:19.851705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:19.851626    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:19.851837 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:19.851794    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:19.853089 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:19.853065    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:37:19.875915 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:19.875865    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podStartSLOduration=2.325797319 podStartE2EDuration="5.875850674s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.550053355s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:37:19.873731757 +0000 UTC m=+1302.129137098" watchObservedRunningTime="2026-07-02 14:37:19.875850674 +0000 UTC m=+1302.131256015"
Jul 02 14:37:20.854660 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:20.854622    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:37:25.860497 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:25.860470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:37:25.860967 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:25.860943    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:37:35.861013 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:35.860976    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:37:45.861916 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:45.861879    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:37:55.861471 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:37:55.861431    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:38:05.861035 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:05.860992    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:38:15.861296 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:15.861257    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:38:25.861760 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:25.861718    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:38:35.861922 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:35.861887    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:38:44.242755 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:44.242723    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"]
Jul 02 14:38:44.243267 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:44.242954    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" containerID="cri-o://6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995" gracePeriod=30
Jul 02 14:38:44.243267 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:44.243000    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kube-rbac-proxy" containerID="cri-o://4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6" gracePeriod=30
Jul 02 14:38:45.149530 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:45.149501    2583 generic.go:356] "Generic (PLEG): container finished" podID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerID="4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6" exitCode=2
Jul 02 14:38:45.149701 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:45.149569    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerDied","Data":"4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6"}
Jul 02 14:38:45.855723 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:45.855681    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 02 14:38:45.861068 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:45.861042    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 02 14:38:48.982401 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:48.982373    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:38:49.034555 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvjvg\" (UniqueName: \"kubernetes.io/projected/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-api-access-qvjvg\") pod \"74a4a633-302b-4a6b-aaff-b061ec2516a6\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") "
Jul 02 14:38:49.034555 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034520    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74a4a633-302b-4a6b-aaff-b061ec2516a6-proxy-tls\") pod \"74a4a633-302b-4a6b-aaff-b061ec2516a6\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") "
Jul 02 14:38:49.034801 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034568    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-rbac-proxy-sar-config\") pod \"74a4a633-302b-4a6b-aaff-b061ec2516a6\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") "
Jul 02 14:38:49.034801 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034600    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-cabundle-cert\") pod \"74a4a633-302b-4a6b-aaff-b061ec2516a6\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") "
Jul 02 14:38:49.034801 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034623    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a4a633-302b-4a6b-aaff-b061ec2516a6-kserve-provision-location\") pod \"74a4a633-302b-4a6b-aaff-b061ec2516a6\" (UID: \"74a4a633-302b-4a6b-aaff-b061ec2516a6\") "
Jul 02 14:38:49.034979 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034955    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74a4a633-302b-4a6b-aaff-b061ec2516a6" (UID: "74a4a633-302b-4a6b-aaff-b061ec2516a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:38:49.035033 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034993    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a4a633-302b-4a6b-aaff-b061ec2516a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74a4a633-302b-4a6b-aaff-b061ec2516a6" (UID: "74a4a633-302b-4a6b-aaff-b061ec2516a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:38:49.035033 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.034994    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74a4a633-302b-4a6b-aaff-b061ec2516a6" (UID: "74a4a633-302b-4a6b-aaff-b061ec2516a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:38:49.036555 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.036534    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a4a633-302b-4a6b-aaff-b061ec2516a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74a4a633-302b-4a6b-aaff-b061ec2516a6" (UID: "74a4a633-302b-4a6b-aaff-b061ec2516a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:38:49.036616 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.036599    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-api-access-qvjvg" (OuterVolumeSpecName: "kube-api-access-qvjvg") pod "74a4a633-302b-4a6b-aaff-b061ec2516a6" (UID: "74a4a633-302b-4a6b-aaff-b061ec2516a6"). InnerVolumeSpecName "kube-api-access-qvjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:38:49.135949 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.135914    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvjvg\" (UniqueName: \"kubernetes.io/projected/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-api-access-qvjvg\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:38:49.135949 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.135943    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74a4a633-302b-4a6b-aaff-b061ec2516a6-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:38:49.135949 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.135954    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:38:49.135949 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.135963    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74a4a633-302b-4a6b-aaff-b061ec2516a6-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:38:49.136230 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.135972    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74a4a633-302b-4a6b-aaff-b061ec2516a6-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:38:49.164511 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.164471    2583 generic.go:356] "Generic (PLEG): container finished" podID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerID="6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995" exitCode=0
Jul 02 14:38:49.164660 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.164530    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"
Jul 02 14:38:49.164660 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.164556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerDied","Data":"6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995"}
Jul 02 14:38:49.164660 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.164614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r" event={"ID":"74a4a633-302b-4a6b-aaff-b061ec2516a6","Type":"ContainerDied","Data":"078891098c8364eaa67059557ead4e142eedd01370189689ea6cdb24c14d649e"}
Jul 02 14:38:49.164660 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.164640    2583 scope.go:117] "RemoveContainer" containerID="4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6"
Jul 02 14:38:49.172469 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.172449    2583 scope.go:117] "RemoveContainer" containerID="6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995"
Jul 02 14:38:49.182366 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.182346    2583 scope.go:117] "RemoveContainer" containerID="b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d"
Jul 02 14:38:49.186209 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.186186    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"]
Jul 02 14:38:49.189889 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.189870    2583 scope.go:117] "RemoveContainer" containerID="4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6"
Jul 02 14:38:49.190175 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:38:49.190155    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6\": container with ID starting with 4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6 not found: ID does not exist" containerID="4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6"
Jul 02 14:38:49.190232 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.190185    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6"} err="failed to get container status \"4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6\": rpc error: code = NotFound desc = could not find container \"4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6\": container with ID starting with 4e80503b42733f16362294740213abb8eb62b2f59410ca7cfdf7e5296b7d25e6 not found: ID does not exist"
Jul 02 14:38:49.190232 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.190205    2583 scope.go:117] "RemoveContainer" containerID="6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995"
Jul 02 14:38:49.190424 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:38:49.190410    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995\": container with ID starting with 6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995 not found: ID does not exist" containerID="6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995"
Jul 02 14:38:49.190466 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.190428    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995"} err="failed to get container status \"6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995\": rpc error: code = NotFound desc = could not find container \"6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995\": container with ID starting with 6d0d15c6b6f06f77019723913015cfe0af5f02afc80368dc4433c43ed5450995 not found: ID does not exist"
Jul 02 14:38:49.190466 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.190441    2583 scope.go:117] "RemoveContainer" containerID="b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d"
Jul 02 14:38:49.190925 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:38:49.190903    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d\": container with ID starting with b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d not found: ID does not exist" containerID="b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d"
Jul 02 14:38:49.191017 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.190942    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d"} err="failed to get container status \"b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d\": rpc error: code = NotFound desc = could not find container \"b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d\": container with ID starting with b59e5e753b2225dddfa0b364bf388a648ac2c8f881b34a0ecea55764c663ec8d not found: ID does not exist"
Jul 02 14:38:49.192554 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.192532    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-5fcfc-predictor-6bc69857f-rtl7r"]
Jul 02 14:38:49.405585 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.405501    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"]
Jul 02 14:38:49.405876 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.405860    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kube-rbac-proxy"
Jul 02 14:38:49.405966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.405879    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kube-rbac-proxy"
Jul 02 14:38:49.405966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.405900    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="storage-initializer"
Jul 02 14:38:49.405966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.405909    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="storage-initializer"
Jul 02 14:38:49.405966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.405926    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container"
Jul 02 14:38:49.405966 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.405935    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container"
Jul 02 14:38:49.406216 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.406022    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kube-rbac-proxy"
Jul 02 14:38:49.406216 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.406037    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" containerName="kserve-container"
Jul 02 14:38:49.410297 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.410274    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.412959 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.412931    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 14:38:49.413105 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.413064    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 14:38:49.413185 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.413100    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 14:38:49.413185 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.413120    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-6f2af-predictor-serving-cert\""
Jul 02 14:38:49.413185 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.413177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6qw4\""
Jul 02 14:38:49.413428 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.413412    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-6f2af-kube-rbac-proxy-sar-config\""
Jul 02 14:38:49.413477 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.413436    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 14:38:49.421842 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.421817    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"]
Jul 02 14:38:49.538536 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.538502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dcd585-8c92-4b29-b754-5b176b678d73-kserve-provision-location\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.538757 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.538542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dcd585-8c92-4b29-b754-5b176b678d73-proxy-tls\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.538757 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.538614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-cabundle-cert\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.538757 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.538710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.538757 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.538749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btmzp\" (UniqueName: \"kubernetes.io/projected/92dcd585-8c92-4b29-b754-5b176b678d73-kube-api-access-btmzp\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.639452 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.639414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dcd585-8c92-4b29-b754-5b176b678d73-kserve-provision-location\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.639452 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.639453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dcd585-8c92-4b29-b754-5b176b678d73-proxy-tls\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.639674 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.639586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-cabundle-cert\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.639715 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.639683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.639749 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.639715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmzp\" (UniqueName: \"kubernetes.io/projected/92dcd585-8c92-4b29-b754-5b176b678d73-kube-api-access-btmzp\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.639917 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.639897    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dcd585-8c92-4b29-b754-5b176b678d73-kserve-provision-location\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.640255 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.640236    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-cabundle-cert\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.640342 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.640311    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.642034 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.642015    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dcd585-8c92-4b29-b754-5b176b678d73-proxy-tls\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.648982 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.648963    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmzp\" (UniqueName: \"kubernetes.io/projected/92dcd585-8c92-4b29-b754-5b176b678d73-kube-api-access-btmzp\") pod \"raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.721010 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.720967    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:49.850741 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:49.850716    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"]
Jul 02 14:38:49.853192 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:38:49.853165    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dcd585_8c92_4b29_b754_5b176b678d73.slice/crio-0c49e042a67c9cce1bc3cb01836aaca32c11174c6cc3bdaaf63310975df514ea WatchSource:0}: Error finding container 0c49e042a67c9cce1bc3cb01836aaca32c11174c6cc3bdaaf63310975df514ea: Status 404 returned error can't find the container with id 0c49e042a67c9cce1bc3cb01836aaca32c11174c6cc3bdaaf63310975df514ea
Jul 02 14:38:50.148946 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:50.148846    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a4a633-302b-4a6b-aaff-b061ec2516a6" path="/var/lib/kubelet/pods/74a4a633-302b-4a6b-aaff-b061ec2516a6/volumes"
Jul 02 14:38:50.169554 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:50.169526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerStarted","Data":"0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317"}
Jul 02 14:38:50.169713 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:50.169588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerStarted","Data":"0c49e042a67c9cce1bc3cb01836aaca32c11174c6cc3bdaaf63310975df514ea"}
Jul 02 14:38:54.185387 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:54.185303    2583 generic.go:356] "Generic (PLEG): container finished" podID="92dcd585-8c92-4b29-b754-5b176b678d73" containerID="0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317" exitCode=0
Jul 02 14:38:54.185387 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:54.185336    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerDied","Data":"0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317"}
Jul 02 14:38:55.190346 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:55.190308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerStarted","Data":"47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e"}
Jul 02 14:38:55.190705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:55.190398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerStarted","Data":"5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77"}
Jul 02 14:38:55.190705 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:55.190594    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:55.212300 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:55.212241    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podStartSLOduration=2.415328244 podStartE2EDuration="6.212223946s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.796895702s" isStatefulPod=false podCreationTimestamp="2026-07-02 14:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:38:55.211462646 +0000 UTC m=+1397.466867988" watchObservedRunningTime="2026-07-02 14:38:55.212223946 +0000 UTC m=+1397.467629288"
Jul 02 14:38:56.194557 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:56.194528    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:38:56.195971 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:56.195941    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:38:57.198033 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:38:57.198001    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:39:02.202456 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:39:02.202424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:39:02.203036 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:39:02.202936    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:39:12.203455 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:39:12.203420    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:39:22.203599 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:39:22.203558    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:39:32.203905 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:39:32.203816    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:39:42.203154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:39:42.203109    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:39:52.203211 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:39:52.203171    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:40:02.203423 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:02.203382    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:40:12.148151 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:12.148123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:40:19.528225 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:19.528188    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"]
Jul 02 14:40:19.528686 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:19.528530    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" containerID="cri-o://5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77" gracePeriod=30
Jul 02 14:40:19.528686 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:19.528615    2583 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kube-rbac-proxy" containerID="cri-o://47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e" gracePeriod=30
Jul 02 14:40:20.485876 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:20.485840    2583 generic.go:356] "Generic (PLEG): container finished" podID="92dcd585-8c92-4b29-b754-5b176b678d73" containerID="47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e" exitCode=2
Jul 02 14:40:20.486045 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:20.485895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerDied","Data":"47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e"}
Jul 02 14:40:22.144234 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:22.144195    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 02 14:40:22.198969 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:22.198932    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 02 14:40:24.072225 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.072200    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:40:24.128907 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.128873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dcd585-8c92-4b29-b754-5b176b678d73-kserve-provision-location\") pod \"92dcd585-8c92-4b29-b754-5b176b678d73\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") "
Jul 02 14:40:24.129106 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.128920    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dcd585-8c92-4b29-b754-5b176b678d73-proxy-tls\") pod \"92dcd585-8c92-4b29-b754-5b176b678d73\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") "
Jul 02 14:40:24.129106 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.128980    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btmzp\" (UniqueName: \"kubernetes.io/projected/92dcd585-8c92-4b29-b754-5b176b678d73-kube-api-access-btmzp\") pod \"92dcd585-8c92-4b29-b754-5b176b678d73\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") "
Jul 02 14:40:24.129106 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.129005    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-cabundle-cert\") pod \"92dcd585-8c92-4b29-b754-5b176b678d73\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") "
Jul 02 14:40:24.129106 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.129050    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-kube-rbac-proxy-sar-config\") pod \"92dcd585-8c92-4b29-b754-5b176b678d73\" (UID: \"92dcd585-8c92-4b29-b754-5b176b678d73\") "
Jul 02 14:40:24.129677 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.129289    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dcd585-8c92-4b29-b754-5b176b678d73-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92dcd585-8c92-4b29-b754-5b176b678d73" (UID: "92dcd585-8c92-4b29-b754-5b176b678d73"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 14:40:24.129677 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.129452    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "92dcd585-8c92-4b29-b754-5b176b678d73" (UID: "92dcd585-8c92-4b29-b754-5b176b678d73"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:40:24.129677 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.129508    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "92dcd585-8c92-4b29-b754-5b176b678d73" (UID: "92dcd585-8c92-4b29-b754-5b176b678d73"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 14:40:24.131150 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.131126    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dcd585-8c92-4b29-b754-5b176b678d73-kube-api-access-btmzp" (OuterVolumeSpecName: "kube-api-access-btmzp") pod "92dcd585-8c92-4b29-b754-5b176b678d73" (UID: "92dcd585-8c92-4b29-b754-5b176b678d73"). InnerVolumeSpecName "kube-api-access-btmzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 14:40:24.131150 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.131128    2583 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92dcd585-8c92-4b29-b754-5b176b678d73-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "92dcd585-8c92-4b29-b754-5b176b678d73" (UID: "92dcd585-8c92-4b29-b754-5b176b678d73"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 14:40:24.230410 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.230370    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:40:24.230410 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.230402    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92dcd585-8c92-4b29-b754-5b176b678d73-kserve-provision-location\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:40:24.230410 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.230414    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/92dcd585-8c92-4b29-b754-5b176b678d73-proxy-tls\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:40:24.230633 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.230430    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btmzp\" (UniqueName: \"kubernetes.io/projected/92dcd585-8c92-4b29-b754-5b176b678d73-kube-api-access-btmzp\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:40:24.230633 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.230440    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/92dcd585-8c92-4b29-b754-5b176b678d73-cabundle-cert\") on node \"ip-10-0-132-82.ec2.internal\" DevicePath \"\""
Jul 02 14:40:24.501595 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.501560    2583 generic.go:356] "Generic (PLEG): container finished" podID="92dcd585-8c92-4b29-b754-5b176b678d73" containerID="5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77" exitCode=0
Jul 02 14:40:24.501760 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.501614    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"
Jul 02 14:40:24.501760 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.501634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerDied","Data":"5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77"}
Jul 02 14:40:24.501760 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.501692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x" event={"ID":"92dcd585-8c92-4b29-b754-5b176b678d73","Type":"ContainerDied","Data":"0c49e042a67c9cce1bc3cb01836aaca32c11174c6cc3bdaaf63310975df514ea"}
Jul 02 14:40:24.501760 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.501718    2583 scope.go:117] "RemoveContainer" containerID="47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e"
Jul 02 14:40:24.509251 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.509226    2583 scope.go:117] "RemoveContainer" containerID="5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77"
Jul 02 14:40:24.515752 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.515738    2583 scope.go:117] "RemoveContainer" containerID="0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317"
Jul 02 14:40:24.519646 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.519625    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"]
Jul 02 14:40:24.523257 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.523242    2583 scope.go:117] "RemoveContainer" containerID="47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e"
Jul 02 14:40:24.523534 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:40:24.523510    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e\": container with ID starting with 47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e not found: ID does not exist" containerID="47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e"
Jul 02 14:40:24.523587 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.523546    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e"} err="failed to get container status \"47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e\": rpc error: code = NotFound desc = could not find container \"47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e\": container with ID starting with 47cd9b0d44ab5aa6abaa0cd43074ee1370a0c944028eeb57156fc6cf2f81777e not found: ID does not exist"
Jul 02 14:40:24.523587 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.523570    2583 scope.go:117] "RemoveContainer" containerID="5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77"
Jul 02 14:40:24.523837 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:40:24.523820    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77\": container with ID starting with 5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77 not found: ID does not exist" containerID="5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77"
Jul 02 14:40:24.523895 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.523843    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77"} err="failed to get container status \"5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77\": rpc error: code = NotFound desc = could not find container \"5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77\": container with ID starting with 5f776ac4ef8537b2dbb80ef1a412515d431ab4e84590a39ed5cdf5cd920a1f77 not found: ID does not exist"
Jul 02 14:40:24.523895 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.523866    2583 scope.go:117] "RemoveContainer" containerID="0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317"
Jul 02 14:40:24.523978 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.523913    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-6f2af-predictor-94f4f45dc-f7l8x"]
Jul 02 14:40:24.524079 ip-10-0-132-82 kubenswrapper[2583]: E0702 14:40:24.524062    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317\": container with ID starting with 0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317 not found: ID does not exist" containerID="0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317"
Jul 02 14:40:24.524118 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:24.524084    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317"} err="failed to get container status \"0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317\": rpc error: code = NotFound desc = could not find container \"0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317\": container with ID starting with 0a1cf86dfd5616b000a30fded2c87c2178a602c8ddee33dafdd80d67145db317 not found: ID does not exist"
Jul 02 14:40:26.148246 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:26.148219    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" path="/var/lib/kubelet/pods/92dcd585-8c92-4b29-b754-5b176b678d73/volumes"
Jul 02 14:40:38.172211 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:38.172183    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:40:38.176413 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:38.176390    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:40:58.110121 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:58.110021    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-88xc5_2aef6263-bcad-4789-89d3-c63178657d4f/global-pull-secret-syncer/0.log"
Jul 02 14:40:58.323076 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:58.323047    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dw2fn_68692d51-6050-407b-8292-af53fb2a0b90/konnectivity-agent/0.log"
Jul 02 14:40:58.344310 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:40:58.344284    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-82.ec2.internal_b6f86f8428c86df58f1aca64ecd7b715/haproxy/0.log"
Jul 02 14:41:03.080504 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:03.080478    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j4p6q_75597c68-a05f-409b-b0e9-bea48443afe2/node-exporter/0.log"
Jul 02 14:41:03.100534 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:03.100508    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j4p6q_75597c68-a05f-409b-b0e9-bea48443afe2/kube-rbac-proxy/0.log"
Jul 02 14:41:03.120866 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:03.120841    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j4p6q_75597c68-a05f-409b-b0e9-bea48443afe2/init-textfile/0.log"
Jul 02 14:41:06.479684 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479648    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"]
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479912    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="storage-initializer"
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479925    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="storage-initializer"
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479936    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container"
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479942    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container"
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479951    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kube-rbac-proxy"
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479957    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kube-rbac-proxy"
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.479999    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kube-rbac-proxy"
Jul 02 14:41:06.480060 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.480007    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="92dcd585-8c92-4b29-b754-5b176b678d73" containerName="kserve-container"
Jul 02 14:41:06.483933 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.483915    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.486748 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.486728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52qrt\"/\"openshift-service-ca.crt\""
Jul 02 14:41:06.487835 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.487817    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-52qrt\"/\"default-dockercfg-mpkrj\""
Jul 02 14:41:06.487955 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.487841    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-52qrt\"/\"kube-root-ca.crt\""
Jul 02 14:41:06.495575 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.495555    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"]
Jul 02 14:41:06.540593 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.540561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-podres\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.540593 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.540596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77hb\" (UniqueName: \"kubernetes.io/projected/fc7e24f4-3fa3-4142-b04d-19805b59a725-kube-api-access-r77hb\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.540825 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.540615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-lib-modules\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.540825 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.540695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-sys\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.540825 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.540722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-proc\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641037 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.640997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-podres\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641037 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77hb\" (UniqueName: \"kubernetes.io/projected/fc7e24f4-3fa3-4142-b04d-19805b59a725-kube-api-access-r77hb\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641230 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-lib-modules\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641230 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-sys\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641230 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641153    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-podres\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641230 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-proc\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641230 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641201    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-lib-modules\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641230 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641220    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-sys\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.641447 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.641252    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc7e24f4-3fa3-4142-b04d-19805b59a725-proc\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.649735 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.649706    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77hb\" (UniqueName: \"kubernetes.io/projected/fc7e24f4-3fa3-4142-b04d-19805b59a725-kube-api-access-r77hb\") pod \"perf-node-gather-daemonset-9tn66\" (UID: \"fc7e24f4-3fa3-4142-b04d-19805b59a725\") " pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.793560 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.793443    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:06.921084 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:06.921052    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"]
Jul 02 14:41:06.923788 ip-10-0-132-82 kubenswrapper[2583]: W0702 14:41:06.923745    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfc7e24f4_3fa3_4142_b04d_19805b59a725.slice/crio-842be8040d23c74007713d195ba1d2f4fe47479b6edeafb25c95690a8dac2c71 WatchSource:0}: Error finding container 842be8040d23c74007713d195ba1d2f4fe47479b6edeafb25c95690a8dac2c71: Status 404 returned error can't find the container with id 842be8040d23c74007713d195ba1d2f4fe47479b6edeafb25c95690a8dac2c71
Jul 02 14:41:07.649179 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:07.649143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66" event={"ID":"fc7e24f4-3fa3-4142-b04d-19805b59a725","Type":"ContainerStarted","Data":"df095338aa7377c820816b71861e382ee2a100de33d2e2a02f8e5f8926e8bb62"}
Jul 02 14:41:07.649556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:07.649195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66" event={"ID":"fc7e24f4-3fa3-4142-b04d-19805b59a725","Type":"ContainerStarted","Data":"842be8040d23c74007713d195ba1d2f4fe47479b6edeafb25c95690a8dac2c71"}
Jul 02 14:41:07.649556 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:07.649256    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:07.666920 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:07.666828    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66" podStartSLOduration=1.66681392 podStartE2EDuration="1.66681392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 14:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 14:41:07.666100768 +0000 UTC m=+1529.921506122" watchObservedRunningTime="2026-07-02 14:41:07.66681392 +0000 UTC m=+1529.922219258"
Jul 02 14:41:07.711886 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:07.711858    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gxzwx_e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a/dns/0.log"
Jul 02 14:41:07.735934 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:07.735907    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gxzwx_e76f9df9-afbd-4cef-8eb1-6e5f1bd6ab1a/kube-rbac-proxy/0.log"
Jul 02 14:41:07.787999 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:07.787966    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5tl5_292e4555-5517-496c-b83a-b1e4090e0579/dns-node-resolver/0.log"
Jul 02 14:41:08.317979 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:08.317950    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-llvbz_36221c70-cb82-48b5-8d8c-d6babeccf3b5/node-ca/0.log"
Jul 02 14:41:09.592424 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:09.592390    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rw985_87f5b14c-698e-466b-a140-08b301139651/serve-healthcheck-canary/0.log"
Jul 02 14:41:10.121278 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:10.121245    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-twj2v_8c165837-8308-4e76-92e8-a226d81bb246/kube-rbac-proxy/0.log"
Jul 02 14:41:10.141259 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:10.141234    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-twj2v_8c165837-8308-4e76-92e8-a226d81bb246/exporter/0.log"
Jul 02 14:41:10.166435 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:10.166411    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-twj2v_8c165837-8308-4e76-92e8-a226d81bb246/extractor/0.log"
Jul 02 14:41:12.292056 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:12.292011    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7b9c77fff9-b6mdl_464a6731-0562-4136-9441-84c9023d76a8/manager/0.log"
Jul 02 14:41:12.334412 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:12.334392    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-6x7tz_6954789d-6e1d-4ac0-a625-832189b39a6e/server/0.log"
Jul 02 14:41:12.501958 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:12.501926    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2wcb7_bad2322a-364c-4736-8077-7b40adf28e9a/seaweedfs/0.log"
Jul 02 14:41:13.661342 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:13.661316    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-52qrt/perf-node-gather-daemonset-9tn66"
Jul 02 14:41:16.561367 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:16.561342    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-hltm4_a3ef0e84-bffb-4237-b25e-f2ed433ce809/migrator/0.log"
Jul 02 14:41:16.583867 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:16.583842    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-hltm4_a3ef0e84-bffb-4237-b25e-f2ed433ce809/graceful-termination/0.log"
Jul 02 14:41:18.066919 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.066839    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg5lm_bc51f60a-9cbb-4c55-9df2-faaa95d7822d/kube-multus-additional-cni-plugins/0.log"
Jul 02 14:41:18.091548 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.091525    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg5lm_bc51f60a-9cbb-4c55-9df2-faaa95d7822d/egress-router-binary-copy/0.log"
Jul 02 14:41:18.115180 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.115158    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg5lm_bc51f60a-9cbb-4c55-9df2-faaa95d7822d/cni-plugins/0.log"
Jul 02 14:41:18.135451 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.135422    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg5lm_bc51f60a-9cbb-4c55-9df2-faaa95d7822d/bond-cni-plugin/0.log"
Jul 02 14:41:18.155159 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.155136    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg5lm_bc51f60a-9cbb-4c55-9df2-faaa95d7822d/routeoverride-cni/0.log"
Jul 02 14:41:18.175833 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.175812    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg5lm_bc51f60a-9cbb-4c55-9df2-faaa95d7822d/whereabouts-cni-bincopy/0.log"
Jul 02 14:41:18.196812 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.196786    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gg5lm_bc51f60a-9cbb-4c55-9df2-faaa95d7822d/whereabouts-cni/0.log"
Jul 02 14:41:18.575465 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.575426    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mnwkv_1627f783-176b-4823-acaa-439926a359dc/kube-multus/0.log"
Jul 02 14:41:18.631481 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.631458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4vgmc_46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18/network-metrics-daemon/0.log"
Jul 02 14:41:18.652737 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:18.652713    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4vgmc_46a6dee6-6ab6-4cd8-a2ff-468d19dfaf18/kube-rbac-proxy/0.log"
Jul 02 14:41:19.729473 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.729445    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-controller/0.log"
Jul 02 14:41:19.749000 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.748971    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/0.log"
Jul 02 14:41:19.749154 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.749047    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovn-acl-logging/1.log"
Jul 02 14:41:19.768018 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.767982    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/kube-rbac-proxy-node/0.log"
Jul 02 14:41:19.792178 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.792155    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 14:41:19.812479 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.812452    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/northd/0.log"
Jul 02 14:41:19.837085 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.837057    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/nbdb/0.log"
Jul 02 14:41:19.862806 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.862779    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/sbdb/0.log"
Jul 02 14:41:19.921318 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:19.921289    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2cfk_75a16613-faa9-4c72-ad72-e46f5e821947/ovnkube-controller/0.log"
Jul 02 14:41:21.224401 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:21.224378    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-s62pd_feefb507-74b8-4f61-b823-359ea3ba6462/network-check-target-container/0.log"
Jul 02 14:41:22.144450 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:22.144421    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dmwts_c88b0850-2950-401e-8569-fe7232e027a0/iptables-alerter/0.log"
Jul 02 14:41:22.928344 ip-10-0-132-82 kubenswrapper[2583]: I0702 14:41:22.928312    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-zfq67_c9fb606c-1df9-4108-95d6-bb407a013248/tuned/0.log"