Jul 01 17:14:14.064400 ip-10-0-129-95 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 01 17:14:14.064413 ip-10-0-129-95 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 01 17:14:14.064420 ip-10-0-129-95 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 01 17:14:14.064682 ip-10-0-129-95 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 01 17:14:24.160328 ip-10-0-129-95 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 01 17:14:24.160347 ip-10-0-129-95 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d14022f1398c45938fa2887f2ca033e4 --
Jul 01 17:16:09.470819 ip-10-0-129-95 systemd[1]: Starting Kubernetes Kubelet...
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.877938    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882538    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882553    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882557    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882560    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882563    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882567    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882570    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:16:10.003049 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882572    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:16:09.948735 ip-10-0-129-95 systemd[1]: Started Kubernetes Kubelet.
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882575    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882577    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882581    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882584    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882587    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882590    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882599    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882602    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882605    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882607    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882610    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882613    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882616    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882621    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882625    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882628    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882632    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:16:10.004188 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882636    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882639    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882642    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882645    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882648    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882651    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882653    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882656    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882659    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882661    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882664    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882666    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882669    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882671    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882674    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882676    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882679    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882681    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882684    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882687    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:16:10.013379 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882689    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882692    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882697    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882700    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882704    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882706    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882710    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882713    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882716    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882719    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882721    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882724    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882726    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882731    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882735    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882738    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882741    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882745    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882748    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:16:10.014590 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882750    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882753    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882755    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882758    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882760    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882763    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882765    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882768    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882771    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882773    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882776    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882778    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882781    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882783    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882786    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882789    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882792    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882795    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882797    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882800    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:16:10.015473 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882803    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882806    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882808    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882811    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882813    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882816    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882818    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882821    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882825    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882827    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882830    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882832    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882835    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882837    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.882840    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883579    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883586    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883590    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883595    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883598    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:16:10.016619 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883600    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883603    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883605    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883608    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883611    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883613    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883616    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883619    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883623    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883626    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883629    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883631    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883634    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883636    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883639    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883641    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883644    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883646    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883649    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883651    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:16:10.019398 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883654    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883656    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883659    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883662    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883665    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883667    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883669    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883672    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883675    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883677    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883682    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883685    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883688    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883691    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883694    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883697    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883700    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883703    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883705    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:16:10.033116 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883708    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883713    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883716    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883720    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883723    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883727    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883730    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883733    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883736    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883740    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883743    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883745    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883748    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883751    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883754    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883757    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883760    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883762    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:16:10.034528 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883765    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883768    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883770    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883773    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883776    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883779    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883783    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883785    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883788    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883791    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883793    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883796    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883798    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883801    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883803    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883806    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883808    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883811    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883814    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883816    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:16:10.035423 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883819    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883821    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883824    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883828    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883831    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883833    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883836    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883838    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883840    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883843    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883845    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883848    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883850    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883853    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883855    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883858    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.883860    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884835    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884843    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884846    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:16:10.036243 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884849    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884852    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884855    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884858    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884860    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884863    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884866    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884868    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884871    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884874    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884877    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884880    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884883    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884885    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884887    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884892    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884894    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884897    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884900    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884902    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:16:10.037104 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884905    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884907    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884909    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884912    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884916    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884920    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884923    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884926    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884929    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884934    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884936    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884939    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884942    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884944    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884947    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884951    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884954    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884956    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884959    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:16:10.037930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884963    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884966    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884968    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884972    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884976    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884978    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884981    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884983    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884990    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884993    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884995    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.884998    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885000    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885003    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885005    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885008    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885010    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885013    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885015    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885018    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:16:10.038685 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885020    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885023    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885027    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885030    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885032    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885035    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885037    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885040    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885043    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885045    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885047    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885050    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885052    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885055    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885058    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885060    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885062    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885065    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885068    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885070    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:16:10.039490 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885075    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885077    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885080    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885082    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885085    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885088    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885091    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885093    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885097    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885101    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885103    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885106    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885108    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885111    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885114    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885117    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885119    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885193    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885200    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885208    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 01 17:16:10.040297 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885213    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885217    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885220    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885224    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885229    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885232    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885235    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885238    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885241    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885244    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885251    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885254    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885259    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885262    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885265    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885269    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885272    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885275    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885278    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885281    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885285    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885288    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885291    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 01 17:16:10.041178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885295    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885298    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885301    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885306    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885310    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885313    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885317    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885320    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885330    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885333    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885336    2576 flags.go:64] FLAG: --enable-server="true"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885339    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885344    2576 flags.go:64] FLAG: --event-burst="100"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885347    2576 flags.go:64] FLAG: --event-qps="50"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885350    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885353    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885356    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885360    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885363    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885366    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885369    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885374    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885377    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885380    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885383    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 01 17:16:10.042018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885385    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885388    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885391    2576 flags.go:64] FLAG: --feature-gates=""
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885395    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885398    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885401    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885404    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885407    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885410    2576 flags.go:64] FLAG: --help="false"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885413    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885417    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885420    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885423    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885427    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885430    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885433    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885436    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885438    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885441    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885444    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885447    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885450    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885452    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885455    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 01 17:16:10.042833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885459    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885462    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885465    2576 flags.go:64] FLAG: --lock-file=""
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885467    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885472    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885475    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885480    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885483    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885486    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885488    2576 flags.go:64] FLAG: --logging-format="text"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885491    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885494    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885497    2576 flags.go:64] FLAG: --manifest-url=""
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885500    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885504    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885507    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885511    2576 flags.go:64] FLAG: --max-pods="110"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885514    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885532    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885537    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885541    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885546    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885551    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885554    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885562    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 01 17:16:10.043658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885565    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885569    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885572    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885575    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885581    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885583    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885587    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885589    2576 flags.go:64] FLAG: --port="10250"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885593    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885596    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-063d9ef39080d0a16"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885599    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885603    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885607    2576 flags.go:64] FLAG: --register-node="true"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885610    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885614    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885616    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885619    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885622    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885626    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885629    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885632    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885635    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885638    2576 flags.go:64] FLAG: --runonce="false"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885640    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885643    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 01 17:16:10.044665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885646    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885651    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885653    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885657    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885660    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885663    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885665    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885668    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885671    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885674    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885677    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885680    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885682    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885687    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885690    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885693    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885697    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885699    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885704    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885707    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885710    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885713    2576 flags.go:64] FLAG: --v="2"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885717    2576 flags.go:64] FLAG: --version="false"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885721    2576 flags.go:64] FLAG: --vmodule=""
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885725    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 01 17:16:10.045369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.885728    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885873    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885877    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885880    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885883    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885886    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885888    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885891    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885897    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885900    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885902    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885905    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885908    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885912    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885916    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885919    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885921    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885924    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885926    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:16:10.155866 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885929    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885931    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885934    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885936    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885939    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885942    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885946    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885948    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885951    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885954    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885957    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885960    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885962    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885965    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885968    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885970    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885972    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885975    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885978    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885980    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:16:10.156922 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885982    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885988    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885991    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885994    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.885997    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886000    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886002    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886005    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886007    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886010    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886012    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886015    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886017    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886020    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886022    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886025    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886027    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886030    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886035    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886037    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:16:10.157781 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886040    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886043    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886046    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886049    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886051    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886054    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886056    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886059    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886061    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886064    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886066    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886068    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886071    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886075    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886077    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886080    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886083    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886085    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886088    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886090    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:16:10.158414 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886093    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886095    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886098    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886100    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886103    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886105    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886108    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886110    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886113    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886116    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886120    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886122    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886125    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886127    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886130    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886133    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886135    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886138    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886140    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:16:10.159053 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886143    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.886147    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.886808    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.895122    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.895142    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.895170    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.895176    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895243    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895248    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895251    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895254    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895257    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895261    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895264    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:16:10.159694 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895266    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895269    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895272    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895275    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895277    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895280    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895283    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895285    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895288    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895291    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895293    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895296    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895298    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895301    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895304    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895307    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895310    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895313    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895315    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895317    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:16:10.160139 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895320    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895323    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895326    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895329    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895332    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895334    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895337    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895340    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895342    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895345    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895347    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895350    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895353    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895355    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895358    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895360    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895363    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895365    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895368    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895370    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:16:10.160782 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895373    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895375    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895378    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895380    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895383    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895385    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895388    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895391    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895393    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895395    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895398    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895401    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895403    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895408    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895411    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895415    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895418    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895422    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895426    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895429    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:16:10.161315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895432    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895435    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895437    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895440    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895443    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895446    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895449    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895451    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895454    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895457    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895460    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895463    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895466    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895470    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895472    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895475    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895478    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895481    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895483    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:16:10.161955 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895486    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895488    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895491    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895493    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895496    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895498    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895502    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895504    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895508    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895511    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895514    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895535    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895540    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.895545    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 17:16:10.162460 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895693    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895698    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895701    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895704    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895708    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895711    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895714    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895716    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895719    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895722    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895725    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895728    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895731    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895733    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895737    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895739    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895742    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895745    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895747    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 01 17:16:10.163055 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895750    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895752    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895755    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895758    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895761    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895767    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895770    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895773    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895778    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895780    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895783    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895786    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895789    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895792    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895794    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895796    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895799    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895802    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895805    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 01 17:16:10.163709 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895808    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895812    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895814    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895817    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895820    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895822    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895825    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895828    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895831    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895833    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895836    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895838    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895841    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895843    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895846    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895848    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895850    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895853    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895857    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895860    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 01 17:16:10.164420 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895862    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895866    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895869    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895871    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895874    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895876    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895879    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895882    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895884    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895887    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895890    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895892    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895894    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895897    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895899    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895909    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895912    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895915    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895917    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895920    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 01 17:16:10.165120 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895923    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895928    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895931    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895934    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895937    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895939    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895942    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895945    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895947    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895950    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895954    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895956    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895959    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895964    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895967    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895969    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895972    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895975    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895977    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895980    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 01 17:16:10.165812 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:09.895982    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.895987    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.896116    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.898293    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.899803    2576 server.go:1019] "Starting client certificate rotation"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.899914    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.899953    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.902370    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.903571    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.914987    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.917134    2576 log.go:25] "Validated CRI v1 image API"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.917954    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 01 17:16:10.166351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.920300    2576 fs.go:135] Filesystem UUIDs: map[5edb7142-9d9e-4368-92a3-06c2f76b3e59:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a3ca7683-649d-46a1-b176-cca45af5469b:/dev/nvme0n1p3]
Jul 01 17:16:10.166785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.920315    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.928161    2576 manager.go:216] Machine: {Timestamp:2026-07-01 17:16:09.924410915 +0000 UTC m=+0.330071681 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100074 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f39acdf18af1822d0b80dcdd96e71 SystemUUID:ec2f39ac-df18-af18-22d0-b80dcdd96e71 BootID:d14022f1-398c-4593-8fa2-887f2ca033e4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e5:0d:dd:74:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e5:0d:dd:74:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:a4:4f:8b:69:a6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.928265    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.928356    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.931580    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.931610    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-95.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.932349    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.932360    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 01 17:16:10.166865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.932368    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.932561    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.933817    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.934615    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.937494    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.937628    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.937656    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.937666    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.937680    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.937689    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.937697    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.938077    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.938102    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.940393    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942057    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942093    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942808    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942832    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942840    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942848    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942860    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942868    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942890    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942909    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942920    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942928    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942951    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.942962    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.943763    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.943774    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.947721    2576 server.go:1268] "Started kubelet"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.947832    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.947953    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.947971    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.948422    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-95.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.948610    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.948741    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.949361    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.950585    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 01 17:16:10.167185 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.953183    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.955048    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.955061    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.955114    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.955672    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.955690    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.955822    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.955831    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.955875    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.956001    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.956014    2576 factory.go:55] Registering systemd factory
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.956023    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.956243    2576 factory.go:153] Registering CRI-O factory
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.956260    2576 factory.go:223] Registration of the crio container factory successfully
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.956280    2576 factory.go:103] Registering Raw factory
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.955600    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-95.ec2.internal.18be39e7bc7ad4ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-95.ec2.internal,UID:ip-10-0-129-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-95.ec2.internal,},FirstTimestamp:2026-07-01 17:16:09.94769019 +0000 UTC m=+0.353350958,LastTimestamp:2026-07-01 17:16:09.94769019 +0000 UTC m=+0.353350958,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-95.ec2.internal,}"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.956763    2576 manager.go:1196] Started watching for new ooms in manager
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.957199    2576 manager.go:349] Starting recovery of all containers
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.961039    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.961145    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.967994    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.970926    2576 manager.go:354] Recovery completed
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.975623    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.977461    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.977496    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.977508    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.978062    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.978072    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.978077    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.978090    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.978754    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.978781    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.978796    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.979253    2576 policy_none.go:49] "None policy: Start"
Jul 01 17:16:10.171090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.979267    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.979277    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.979750    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-95.ec2.internal.18be39e7be416a8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-95.ec2.internal,UID:ip-10-0-129-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-95.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-95.ec2.internal,},FirstTimestamp:2026-07-01 17:16:09.97748187 +0000 UTC m=+0.383142635,LastTimestamp:2026-07-01 17:16:09.97748187 +0000 UTC m=+0.383142635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-95.ec2.internal,}"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.980281    2576 policy_none.go:47] "Start"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.988307    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mrjb2"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:09.993203    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-95.ec2.internal.18be39e7be41b61b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-95.ec2.internal,UID:ip-10-0-129-95.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-95.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-95.ec2.internal,},FirstTimestamp:2026-07-01 17:16:09.977501211 +0000 UTC m=+0.383161978,LastTimestamp:2026-07-01 17:16:09.977501211 +0000 UTC m=+0.383161978,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-95.ec2.internal,}"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:09.994675    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mrjb2"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.040391    2576 manager.go:337] "Starting Device Plugin manager"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.040432    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.040448    2576 server.go:87] "Starting device plugin registration server"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.040752    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.040828    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.040944    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.041032    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.041041    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.041629    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.041676    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.042359    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.042398    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.042419    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.042449    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.044276    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.161229    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"]
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.161478    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.161276    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.162923    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:16:10.172224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.162945    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.162974    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.162991    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.162953    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.163105    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.163128    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.164346    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.164429    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.164453    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.165276    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.165282    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.165296    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.165301    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.165320    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.165354    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.165461    2576 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-95.ec2.internal\" not found" node="ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.166612    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.166632    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.172006    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.172021    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-95.ec2.internal\": node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.172760    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientMemory"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.172785    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasNoDiskPressure"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.172809    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeHasSufficientPID"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.197478    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.222231    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-95.ec2.internal\" not found" node="ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.244878 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.227915    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-95.ec2.internal\" not found" node="ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.261404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50c0f800b9ed31df61aea932adc45e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"50c0f800b9ed31df61aea932adc45e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.261431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2c3d4ff9542796e61a3b334a5839879-config\") pod \"kube-apiserver-proxy-ip-10-0-129-95.ec2.internal\" (UID: \"a2c3d4ff9542796e61a3b334a5839879\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.261447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50c0f800b9ed31df61aea932adc45e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"50c0f800b9ed31df61aea932adc45e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.298312    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.361746    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50c0f800b9ed31df61aea932adc45e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"50c0f800b9ed31df61aea932adc45e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.361787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50c0f800b9ed31df61aea932adc45e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"50c0f800b9ed31df61aea932adc45e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.361805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2c3d4ff9542796e61a3b334a5839879-config\") pod \"kube-apiserver-proxy-ip-10-0-129-95.ec2.internal\" (UID: \"a2c3d4ff9542796e61a3b334a5839879\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.361841    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50c0f800b9ed31df61aea932adc45e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"50c0f800b9ed31df61aea932adc45e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.361851    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/50c0f800b9ed31df61aea932adc45e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal\" (UID: \"50c0f800b9ed31df61aea932adc45e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.374205 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.361880    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2c3d4ff9542796e61a3b334a5839879-config\") pod \"kube-apiserver-proxy-ip-10-0-129-95.ec2.internal\" (UID: \"a2c3d4ff9542796e61a3b334a5839879\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.492145 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.398865    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.606473 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.500938    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.606473 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.526413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.606473 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.532041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:10.606473 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.601484    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.731379 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.703327    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.839167 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.804546    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:10.946280 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.899415    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 01 17:16:10.946280 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.899636    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 01 17:16:10.946280 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:10.905011    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:11.040287 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.998398    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-30 17:11:10 +0000 UTC" deadline="2028-04-05 09:40:44.634476856 +0000 UTC"
Jul 01 17:16:11.040287 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:10.998459    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15448h24m33.636021945s"
Jul 01 17:16:11.040287 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:11.005993    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-95.ec2.internal\" not found"
Jul 01 17:16:11.040287 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.024144    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:16:11.043389 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.043172    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:16:11.140800 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.056199    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:11.140800 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.070266    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 17:16:11.140800 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.072650    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal"
Jul 01 17:16:11.140800 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.082586    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 01 17:16:11.363054 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:11.280385    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c0f800b9ed31df61aea932adc45e70.slice/crio-2b3efb6e7dc4a5a5ad532b2e9e19b7944a8a57e5d8103d12dfe8e57741e446ef WatchSource:0}: Error finding container 2b3efb6e7dc4a5a5ad532b2e9e19b7944a8a57e5d8103d12dfe8e57741e446ef: Status 404 returned error can't find the container with id 2b3efb6e7dc4a5a5ad532b2e9e19b7944a8a57e5d8103d12dfe8e57741e446ef
Jul 01 17:16:11.363054 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.285135    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:16:11.363054 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.290737    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.939809    2576 apiserver.go:52] "Watching apiserver"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.947865    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.949096    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp","openshift-cluster-node-tuning-operator/tuned-5v4vn","openshift-dns/node-resolver-lm9pd","openshift-image-registry/node-ca-jcctl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal","openshift-network-diagnostics/network-check-target-nqm5t","kube-system/global-pull-secret-syncer-psbvf","openshift-multus/multus-additional-cni-plugins-nh6jx","openshift-multus/multus-vczhz","openshift-multus/network-metrics-daemon-tb5ht","openshift-network-operator/iptables-alerter-l9hwb","openshift-ovn-kubernetes/ovnkube-node-qtzvn","kube-system/konnectivity-agent-88pqd"]
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.950540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:11.950620    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.951936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.953809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955649    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955750    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955764    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955786    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955815    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955754    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955984    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.955985    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.956094    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tfpld\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.956283    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.956663    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmjhx\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.956999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.957546    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.958043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sh8fg\""
Jul 01 17:16:11.958228 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.958118    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 01 17:16:11.961265 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.959857    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2n2jp\""
Jul 01 17:16:11.961265 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.960020    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 01 17:16:11.961265 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.960085    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 01 17:16:11.961265 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.960250    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 01 17:16:11.962536 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.962503    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:11.964937 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.964913    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 01 17:16:11.965057 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.965015    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:11.965220 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.965192    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 01 17:16:11.965309 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.965233    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qrxfl\""
Jul 01 17:16:11.966279 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.966261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:11.966385 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:11.966361    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:11.966439 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.966394    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 01 17:16:11.967613 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.967593    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:11.967817 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.967800    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 01 17:16:11.968070 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.968053    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 01 17:16:11.968153 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.968072    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 01 17:16:11.968153 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.968094    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5rqc7\""
Jul 01 17:16:11.968253 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.968163    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 01 17:16:11.968358 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.968342    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 01 17:16:11.968958 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.968941    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vczhz"
Jul 01 17:16:11.969838 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.969727    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 01 17:16:11.970060 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970002    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970232    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970348    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9rjn4\""
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-systemd\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrs5f\" (UniqueName: \"kubernetes.io/projected/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-kube-api-access-rrs5f\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970580    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-host\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvl9v\" (UniqueName: \"kubernetes.io/projected/5a686c1c-ce50-4e13-82d7-2296f4797d30-kube-api-access-gvl9v\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ba3c39-1752-4fac-b0d3-8868786e2412-tmp-dir\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:11.970650    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovnkube-config\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-env-overrides\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-systemd\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a686c1c-ce50-4e13-82d7-2296f4797d30-tmp\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.970785 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-kubelet\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-var-lib-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-log-socket\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchwb\" (UniqueName: \"kubernetes.io/projected/7373fb09-974e-47d2-8064-e91523f0f494-kube-api-access-fchwb\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-kubernetes\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysctl-d\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.970979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-systemd-units\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-var-lib-kubelet\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vd4\" (UniqueName: \"kubernetes.io/projected/c0ba3c39-1752-4fac-b0d3-8868786e2412-kube-api-access-p6vd4\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-slash\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysconfig\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-run\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.971551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7373fb09-974e-47d2-8064-e91523f0f494-serviceca\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-ovn\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-sys\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-tuned\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0ba3c39-1752-4fac-b0d3-8868786e2412-hosts-file\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7373fb09-974e-47d2-8064-e91523f0f494-host\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-run-netns\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-cni-bin\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-cni-netd\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971405    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-lib-modules\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-modprobe-d\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.972237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-etc-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.973811 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-node-log\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.973811 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovnkube-script-lib\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:11.973811 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysctl-conf\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:11.973811 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971672    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:11.973811 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.971970    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 01 17:16:11.973811 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.972216    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hb6zx\""
Jul 01 17:16:11.974092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.973962    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 01 17:16:11.974668 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.974331    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 01 17:16:11.974668 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.974422    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7hz74\""
Jul 01 17:16:11.974668 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.974449    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 01 17:16:11.994387 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:11.994357    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c2hrm"
Jul 01 17:16:12.003661 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.003633    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c2hrm"
Jul 01 17:16:12.046938 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.046827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" event={"ID":"50c0f800b9ed31df61aea932adc45e70","Type":"ContainerStarted","Data":"2b3efb6e7dc4a5a5ad532b2e9e19b7944a8a57e5d8103d12dfe8e57741e446ef"}
Jul 01 17:16:12.056657 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.056630    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 01 17:16:12.072071 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchwb\" (UniqueName: \"kubernetes.io/projected/7373fb09-974e-47d2-8064-e91523f0f494-kube-api-access-fchwb\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:12.072233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97908b6c-1862-4784-bb3a-463d7a4cc872-agent-certs\") pod \"konnectivity-agent-88pqd\" (UID: \"97908b6c-1862-4784-bb3a-463d7a4cc872\") " pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:12.072233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97908b6c-1862-4784-bb3a-463d7a4cc872-konnectivity-ca\") pod \"konnectivity-agent-88pqd\" (UID: \"97908b6c-1862-4784-bb3a-463d7a4cc872\") " pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:12.072233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-kubelet-config\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.072233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-conf-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.072233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw26r\" (UniqueName: \"kubernetes.io/projected/7e3f55f5-2715-4960-8d8f-546f4a38d95b-kube-api-access-bw26r\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.072233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-kubernetes\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysctl-d\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-var-lib-kubelet\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-kubelet\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072311    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-kubernetes\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysconfig\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7373fb09-974e-47d2-8064-e91523f0f494-serviceca\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072368    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-var-lib-kubelet\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6db\" (UniqueName: \"kubernetes.io/projected/e17c5967-c89a-47b9-9d37-3281f0f2b99c-kube-api-access-wg6db\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-device-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-ovn\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-sys\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072451    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysconfig\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysctl-d\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0ba3c39-1752-4fac-b0d3-8868786e2412-hosts-file\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072536    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-ovn\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072593    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-lib-modules\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072646    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0ba3c39-1752-4fac-b0d3-8868786e2412-hosts-file\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-socket-dir-parent\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-daemon-config\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072691    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.072845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072731    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-sys\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-modprobe-d\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-node-log\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-lib-modules\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrs5f\" (UniqueName: \"kubernetes.io/projected/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-kube-api-access-rrs5f\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072859    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7373fb09-974e-47d2-8064-e91523f0f494-serviceca\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-host\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072886    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-node-log\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvl9v\" (UniqueName: \"kubernetes.io/projected/5a686c1c-ce50-4e13-82d7-2296f4797d30-kube-api-access-gvl9v\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-cnibin\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovnkube-config\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.072997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a686c1c-ce50-4e13-82d7-2296f4797d30-tmp\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073218 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-cni-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-var-lib-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-log-socket\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073132    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-modprobe-d\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-var-lib-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-dbus\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073198    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-host\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f67b\" (UniqueName: \"kubernetes.io/projected/f720c753-1293-4ea0-ac00-7b4ca86b573e-kube-api-access-4f67b\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073219    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-log-socket\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-k8s-cni-cncf-io\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-systemd-units\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vd4\" (UniqueName: \"kubernetes.io/projected/c0ba3c39-1752-4fac-b0d3-8868786e2412-kube-api-access-p6vd4\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:12.073723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-system-cni-dir\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e17c5967-c89a-47b9-9d37-3281f0f2b99c-host-slash\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-system-cni-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-slash\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-run\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073462    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073566    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-run\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073599    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovnkube-config\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-slash\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-systemd-units\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-hostroot\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-tuned\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.073971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7373fb09-974e-47d2-8064-e91523f0f494-host\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074045    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7373fb09-974e-47d2-8064-e91523f0f494-host\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-run-netns\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-cni-bin\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-cni-netd\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-socket-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074219    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-cni-bin\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-os-release\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-netns\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074274    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-run-netns\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsk9\" (UniqueName: \"kubernetes.io/projected/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-kube-api-access-vnsk9\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-etc-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.074929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovnkube-script-lib\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074367    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-etc-openvswitch\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysctl-conf\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cnibin\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074426    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-cni-netd\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-registration-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074469    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-sysctl-conf\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-sys-fs\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-cni-binary-copy\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-systemd\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ba3c39-1752-4fac-b0d3-8868786e2412-tmp-dir\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-cni-bin\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-run-systemd\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.075586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-cni-multus\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-multus-certs\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-etc-kubernetes\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-env-overrides\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-systemd\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9g9l\" (UniqueName: \"kubernetes.io/projected/c7ba0dbe-db80-4c86-afc4-545af38d30d0-kube-api-access-p9g9l\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-os-release\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e17c5967-c89a-47b9-9d37-3281f0f2b99c-iptables-alerter-script\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-kubelet\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074909    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ba3c39-1752-4fac-b0d3-8868786e2412-tmp-dir\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074908    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovnkube-script-lib\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074957    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-host-kubelet\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.074974    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-systemd\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.076449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.075095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-env-overrides\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.077088 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.077062    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a686c1c-ce50-4e13-82d7-2296f4797d30-etc-tuned\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.077202 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.077089    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a686c1c-ce50-4e13-82d7-2296f4797d30-tmp\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.077312 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.077299    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-ovn-node-metrics-cert\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.081530 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.081492    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchwb\" (UniqueName: \"kubernetes.io/projected/7373fb09-974e-47d2-8064-e91523f0f494-kube-api-access-fchwb\") pod \"node-ca-jcctl\" (UID: \"7373fb09-974e-47d2-8064-e91523f0f494\") " pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:12.081850 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.081665    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:16:12.081850 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.081685    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:16:12.081850 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.081699    2576 projected.go:196] Error preparing data for projected volume kube-api-access-86qzs for pod openshift-network-diagnostics/network-check-target-nqm5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:12.081850 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.081792    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs podName:a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:12.581771134 +0000 UTC m=+2.987431890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-86qzs" (UniqueName: "kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs") pod "network-check-target-nqm5t" (UID: "a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:12.082095 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.082047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvl9v\" (UniqueName: \"kubernetes.io/projected/5a686c1c-ce50-4e13-82d7-2296f4797d30-kube-api-access-gvl9v\") pod \"tuned-5v4vn\" (UID: \"5a686c1c-ce50-4e13-82d7-2296f4797d30\") " pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.083226 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.083204    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vd4\" (UniqueName: \"kubernetes.io/projected/c0ba3c39-1752-4fac-b0d3-8868786e2412-kube-api-access-p6vd4\") pod \"node-resolver-lm9pd\" (UID: \"c0ba3c39-1752-4fac-b0d3-8868786e2412\") " pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:12.083323 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.083268    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrs5f\" (UniqueName: \"kubernetes.io/projected/c12fa6c4-e35f-4aa2-8b8e-f274469878a9-kube-api-access-rrs5f\") pod \"ovnkube-node-qtzvn\" (UID: \"c12fa6c4-e35f-4aa2-8b8e-f274469878a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.167070 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.167028    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 01 17:16:12.175777 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw26r\" (UniqueName: \"kubernetes.io/projected/7e3f55f5-2715-4960-8d8f-546f4a38d95b-kube-api-access-bw26r\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.175777 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-kubelet\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6db\" (UniqueName: \"kubernetes.io/projected/e17c5967-c89a-47b9-9d37-3281f0f2b99c-kube-api-access-wg6db\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-device-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175872    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-kubelet\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-socket-dir-parent\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-daemon-config\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.176000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.175998    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-cnibin\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176048    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-device-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176063    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-cni-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-socket-dir-parent\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-dbus\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f67b\" (UniqueName: \"kubernetes.io/projected/f720c753-1293-4ea0-ac00-7b4ca86b573e-kube-api-access-4f67b\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-cnibin\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-k8s-cni-cncf-io\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176216    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-dbus\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-system-cni-dir\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.176424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176246    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-system-cni-dir\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e17c5967-c89a-47b9-9d37-3281f0f2b99c-host-slash\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176275    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e17c5967-c89a-47b9-9d37-3281f0f2b99c-host-slash\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-system-cni-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176321    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-system-cni-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-hostroot\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176365    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176370    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-cni-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-socket-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176413    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-k8s-cni-cncf-io\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-os-release\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176467    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-os-release\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176503    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-netns\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-socket-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.176599    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-netns\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsk9\" (UniqueName: \"kubernetes.io/projected/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-kube-api-access-vnsk9\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.176661    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret podName:ff5e36dc-a802-4ffd-99ec-db7ceb2429ba nodeName:}" failed. No retries permitted until 2026-07-01 17:16:12.676641644 +0000 UTC m=+3.082302401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret") pod "global-pull-secret-syncer-psbvf" (UID: "ff5e36dc-a802-4ffd-99ec-db7ceb2429ba") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cnibin\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176704    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-daemon-config\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-registration-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176725    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-sys-fs\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-cni-binary-copy\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-hostroot\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-cni-bin\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.177773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176830    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-registration-dir\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176843    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cni-binary-copy\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-cni-multus\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f720c753-1293-4ea0-ac00-7b4ca86b573e-sys-fs\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-multus-certs\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176904    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-cnibin\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-etc-kubernetes\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.176941    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176947    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-cni-bin\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.176952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9g9l\" (UniqueName: \"kubernetes.io/projected/c7ba0dbe-db80-4c86-afc4-545af38d30d0-kube-api-access-p9g9l\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.176985    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:12.676970571 +0000 UTC m=+3.082631332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-os-release\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177012    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-var-lib-cni-multus\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e17c5967-c89a-47b9-9d37-3281f0f2b99c-iptables-alerter-script\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.178424 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-host-run-multus-certs\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97908b6c-1862-4784-bb3a-463d7a4cc872-agent-certs\") pod \"konnectivity-agent-88pqd\" (UID: \"97908b6c-1862-4784-bb3a-463d7a4cc872\") " pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97908b6c-1862-4784-bb3a-463d7a4cc872-konnectivity-ca\") pod \"konnectivity-agent-88pqd\" (UID: \"97908b6c-1862-4784-bb3a-463d7a4cc872\") " pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177112    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e3f55f5-2715-4960-8d8f-546f4a38d95b-os-release\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-kubelet-config\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-conf-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177168    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-etc-kubernetes\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-kubelet-config\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177236    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-multus-conf-dir\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177279    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e3f55f5-2715-4960-8d8f-546f4a38d95b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-cni-binary-copy\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177701    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e17c5967-c89a-47b9-9d37-3281f0f2b99c-iptables-alerter-script\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.178991 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.177734    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97908b6c-1862-4784-bb3a-463d7a4cc872-konnectivity-ca\") pod \"konnectivity-agent-88pqd\" (UID: \"97908b6c-1862-4784-bb3a-463d7a4cc872\") " pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:12.179693 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.179675    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97908b6c-1862-4784-bb3a-463d7a4cc872-agent-certs\") pod \"konnectivity-agent-88pqd\" (UID: \"97908b6c-1862-4784-bb3a-463d7a4cc872\") " pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:12.189253 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.189226    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw26r\" (UniqueName: \"kubernetes.io/projected/7e3f55f5-2715-4960-8d8f-546f4a38d95b-kube-api-access-bw26r\") pod \"multus-additional-cni-plugins-nh6jx\" (UID: \"7e3f55f5-2715-4960-8d8f-546f4a38d95b\") " pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.189858 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.189825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6db\" (UniqueName: \"kubernetes.io/projected/e17c5967-c89a-47b9-9d37-3281f0f2b99c-kube-api-access-wg6db\") pod \"iptables-alerter-l9hwb\" (UID: \"e17c5967-c89a-47b9-9d37-3281f0f2b99c\") " pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.189858 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.189828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f67b\" (UniqueName: \"kubernetes.io/projected/f720c753-1293-4ea0-ac00-7b4ca86b573e-kube-api-access-4f67b\") pod \"aws-ebs-csi-driver-node-l6vsp\" (UID: \"f720c753-1293-4ea0-ac00-7b4ca86b573e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.190121 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.190096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsk9\" (UniqueName: \"kubernetes.io/projected/f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9-kube-api-access-vnsk9\") pod \"multus-vczhz\" (UID: \"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9\") " pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.190288 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.190267    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9g9l\" (UniqueName: \"kubernetes.io/projected/c7ba0dbe-db80-4c86-afc4-545af38d30d0-kube-api-access-p9g9l\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:12.264417 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.264382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:12.274192 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.274164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jcctl"
Jul 01 17:16:12.274706 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:12.274681    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12fa6c4_e35f_4aa2_8b8e_f274469878a9.slice/crio-1dee4b31bb91e6a1069c4f399d7226e6dcabf95e388b738344f880f204dc793f WatchSource:0}: Error finding container 1dee4b31bb91e6a1069c4f399d7226e6dcabf95e388b738344f880f204dc793f: Status 404 returned error can't find the container with id 1dee4b31bb91e6a1069c4f399d7226e6dcabf95e388b738344f880f204dc793f
Jul 01 17:16:12.282640 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.282617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5v4vn"
Jul 01 17:16:12.282930 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:12.282906    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7373fb09_974e_47d2_8064_e91523f0f494.slice/crio-f9c2e4b1b176f1f4ef289d93631830f2e781be8bacf180e0a3948150afa05ef5 WatchSource:0}: Error finding container f9c2e4b1b176f1f4ef289d93631830f2e781be8bacf180e0a3948150afa05ef5: Status 404 returned error can't find the container with id f9c2e4b1b176f1f4ef289d93631830f2e781be8bacf180e0a3948150afa05ef5
Jul 01 17:16:12.288388 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.288370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lm9pd"
Jul 01 17:16:12.295379 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.295354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:12.302028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.301980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nh6jx"
Jul 01 17:16:12.308737 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.308660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp"
Jul 01 17:16:12.315296 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.315276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vczhz"
Jul 01 17:16:12.320921 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.320900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9hwb"
Jul 01 17:16:12.607315 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:12.607287    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72fd9ff_c4f4_44d7_afb1_c7c8313e08b9.slice/crio-1c6026d43c8c4e7472e2c65128867f1581471fc9cb15f15e2acdfd01c4994f98 WatchSource:0}: Error finding container 1c6026d43c8c4e7472e2c65128867f1581471fc9cb15f15e2acdfd01c4994f98: Status 404 returned error can't find the container with id 1c6026d43c8c4e7472e2c65128867f1581471fc9cb15f15e2acdfd01c4994f98
Jul 01 17:16:12.619773 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:12.619750    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97908b6c_1862_4784_bb3a_463d7a4cc872.slice/crio-2b680977e6d6dc0d06d911f08a294346cb7f5f74b889cf362be11bdb97df37d7 WatchSource:0}: Error finding container 2b680977e6d6dc0d06d911f08a294346cb7f5f74b889cf362be11bdb97df37d7: Status 404 returned error can't find the container with id 2b680977e6d6dc0d06d911f08a294346cb7f5f74b889cf362be11bdb97df37d7
Jul 01 17:16:12.622988 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:12.622958    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17c5967_c89a_47b9_9d37_3281f0f2b99c.slice/crio-9b150cddbd38fd49c83179f5c00ba5398a94a4d7982a0e29769577bc09e59b3b WatchSource:0}: Error finding container 9b150cddbd38fd49c83179f5c00ba5398a94a4d7982a0e29769577bc09e59b3b: Status 404 returned error can't find the container with id 9b150cddbd38fd49c83179f5c00ba5398a94a4d7982a0e29769577bc09e59b3b
Jul 01 17:16:12.623491 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:12.623471    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf720c753_1293_4ea0_ac00_7b4ca86b573e.slice/crio-9aaa7fbd27dce28c347dd9b4be7dac14560f6c2341f5b9761b42656159cc9517 WatchSource:0}: Error finding container 9aaa7fbd27dce28c347dd9b4be7dac14560f6c2341f5b9761b42656159cc9517: Status 404 returned error can't find the container with id 9aaa7fbd27dce28c347dd9b4be7dac14560f6c2341f5b9761b42656159cc9517
Jul 01 17:16:12.680735 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.680660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:12.680903 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.680778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:12.680903 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:12.680834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:12.680903 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.680835    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:12.681055 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.680909    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret podName:ff5e36dc-a802-4ffd-99ec-db7ceb2429ba nodeName:}" failed. No retries permitted until 2026-07-01 17:16:13.680885845 +0000 UTC m=+4.086546620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret") pod "global-pull-secret-syncer-psbvf" (UID: "ff5e36dc-a802-4ffd-99ec-db7ceb2429ba") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:12.681055 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.680944    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:12.681055 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.680957    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:16:12.681055 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.680972    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:16:12.681055 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.680983    2576 projected.go:196] Error preparing data for projected volume kube-api-access-86qzs for pod openshift-network-diagnostics/network-check-target-nqm5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:12.681055 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.681001    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:13.680985875 +0000 UTC m=+4.086646634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:12.681055 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:12.681018    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs podName:a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:13.681006786 +0000 UTC m=+4.086667556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-86qzs" (UniqueName: "kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs") pod "network-check-target-nqm5t" (UID: "a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:12.704910 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:12.704877    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c3d4ff9542796e61a3b334a5839879.slice/crio-840588270f0844af3c464301f62ff258b598c3c9037ad7d395c07a1ac0eaee71 WatchSource:0}: Error finding container 840588270f0844af3c464301f62ff258b598c3c9037ad7d395c07a1ac0eaee71: Status 404 returned error can't find the container with id 840588270f0844af3c464301f62ff258b598c3c9037ad7d395c07a1ac0eaee71
Jul 01 17:16:13.006717 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.006439    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 17:11:11 +0000 UTC" deadline="2027-11-26 14:16:26.713444758 +0000 UTC"
Jul 01 17:16:13.006717 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.006718    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12309h0m13.706734988s"
Jul 01 17:16:13.042878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.042842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:13.043079 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.042993    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:13.051467 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.051430    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9hwb" event={"ID":"e17c5967-c89a-47b9-9d37-3281f0f2b99c","Type":"ContainerStarted","Data":"9b150cddbd38fd49c83179f5c00ba5398a94a4d7982a0e29769577bc09e59b3b"}
Jul 01 17:16:13.053194 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.053168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vczhz" event={"ID":"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9","Type":"ContainerStarted","Data":"1c6026d43c8c4e7472e2c65128867f1581471fc9cb15f15e2acdfd01c4994f98"}
Jul 01 17:16:13.054658 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.054633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jcctl" event={"ID":"7373fb09-974e-47d2-8064-e91523f0f494","Type":"ContainerStarted","Data":"f9c2e4b1b176f1f4ef289d93631830f2e781be8bacf180e0a3948150afa05ef5"}
Jul 01 17:16:13.056409 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.056384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"1dee4b31bb91e6a1069c4f399d7226e6dcabf95e388b738344f880f204dc793f"}
Jul 01 17:16:13.059566 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.059246    2576 generic.go:356] "Generic (PLEG): container finished" podID="50c0f800b9ed31df61aea932adc45e70" containerID="57407a9a16a82bec326ed2649dfdb6bf60faddb4b045848b9afc76a0da8551bc" exitCode=0
Jul 01 17:16:13.059566 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.059339    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" event={"ID":"50c0f800b9ed31df61aea932adc45e70","Type":"ContainerDied","Data":"57407a9a16a82bec326ed2649dfdb6bf60faddb4b045848b9afc76a0da8551bc"}
Jul 01 17:16:13.061390 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.061367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" event={"ID":"a2c3d4ff9542796e61a3b334a5839879","Type":"ContainerStarted","Data":"840588270f0844af3c464301f62ff258b598c3c9037ad7d395c07a1ac0eaee71"}
Jul 01 17:16:13.065551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.065509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lm9pd" event={"ID":"c0ba3c39-1752-4fac-b0d3-8868786e2412","Type":"ContainerStarted","Data":"5ffea2647dc85560758c45e47298fb9187415c44140e532ee8ee5ecd8de11bd3"}
Jul 01 17:16:13.068012 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.067964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5v4vn" event={"ID":"5a686c1c-ce50-4e13-82d7-2296f4797d30","Type":"ContainerStarted","Data":"6f439f7309fdbda5cfe82894de2e91d1b8214c08297d29ee3cb8498fc5071971"}
Jul 01 17:16:13.072467 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.072444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88pqd" event={"ID":"97908b6c-1862-4784-bb3a-463d7a4cc872","Type":"ContainerStarted","Data":"2b680977e6d6dc0d06d911f08a294346cb7f5f74b889cf362be11bdb97df37d7"}
Jul 01 17:16:13.077402 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.075922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerStarted","Data":"48b09aec7e1b33e60dfd33831aaa78c45745ed1fd460302b2638eefbf0c12a8e"}
Jul 01 17:16:13.078922 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.078896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp" event={"ID":"f720c753-1293-4ea0-ac00-7b4ca86b573e","Type":"ContainerStarted","Data":"9aaa7fbd27dce28c347dd9b4be7dac14560f6c2341f5b9761b42656159cc9517"}
Jul 01 17:16:13.689607 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.689325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:13.689607 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.689385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:13.689607 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:13.689439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:13.689932 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.689689    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:16:13.689932 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.689711    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:16:13.689932 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.689724    2576 projected.go:196] Error preparing data for projected volume kube-api-access-86qzs for pod openshift-network-diagnostics/network-check-target-nqm5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:13.689932 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.689781    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs podName:a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:15.689762264 +0000 UTC m=+6.095423022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-86qzs" (UniqueName: "kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs") pod "network-check-target-nqm5t" (UID: "a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:13.690399 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.690169    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:13.690399 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.690226    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret podName:ff5e36dc-a802-4ffd-99ec-db7ceb2429ba nodeName:}" failed. No retries permitted until 2026-07-01 17:16:15.690207279 +0000 UTC m=+6.095868049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret") pod "global-pull-secret-syncer-psbvf" (UID: "ff5e36dc-a802-4ffd-99ec-db7ceb2429ba") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:13.690399 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.690284    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:13.690399 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:13.690316    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:15.690305174 +0000 UTC m=+6.095965934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:14.007542 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:14.007485    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-30 17:11:11 +0000 UTC" deadline="2028-02-03 11:16:25.749063259 +0000 UTC"
Jul 01 17:16:14.007542 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:14.007541    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13962h0m11.741527653s"
Jul 01 17:16:14.043905 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:14.043359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:14.043905 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:14.043494    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:14.044337 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:14.044291    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:14.049150 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:14.045780    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:14.098858 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:14.098071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" event={"ID":"50c0f800b9ed31df61aea932adc45e70","Type":"ContainerStarted","Data":"393657c401db92ec989e0559407da693a206e16269d2f4fb7b7b8a2c8991189a"}
Jul 01 17:16:14.117196 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:14.117147    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-95.ec2.internal" podStartSLOduration=3.117097696 podStartE2EDuration="3.117097696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:14.1166339 +0000 UTC m=+4.522294677" watchObservedRunningTime="2026-07-01 17:16:14.117097696 +0000 UTC m=+4.522792105"
Jul 01 17:16:15.042634 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:15.042601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:15.042827 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.042747    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:15.712802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:15.712879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:15.712916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713042    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713103    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret podName:ff5e36dc-a802-4ffd-99ec-db7ceb2429ba nodeName:}" failed. No retries permitted until 2026-07-01 17:16:19.713085262 +0000 UTC m=+10.118746018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret") pod "global-pull-secret-syncer-psbvf" (UID: "ff5e36dc-a802-4ffd-99ec-db7ceb2429ba") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713492    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713556    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:19.713539994 +0000 UTC m=+10.119200755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713638    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713657    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713669    2576 projected.go:196] Error preparing data for projected volume kube-api-access-86qzs for pod openshift-network-diagnostics/network-check-target-nqm5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:15.713753 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:15.713702    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs podName:a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:19.713691414 +0000 UTC m=+10.119352175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-86qzs" (UniqueName: "kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs") pod "network-check-target-nqm5t" (UID: "a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:16.045901 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:16.043685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:16.045901 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:16.043702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:16.045901 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:16.043816    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:16.045901 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:16.043889    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:17.043184 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:17.043151    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:17.043681 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:17.043309    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:18.042722 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:18.042680    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:18.042904 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:18.042823    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:18.043077 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:18.043051    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:18.043199 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:18.043166    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:19.043680 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:19.043583    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:19.044179 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.043730    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:19.749860 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:19.749763    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:19.749860 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:19.749832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:19.750075 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:19.749875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:19.750075 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750013    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:19.750075 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750042    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:16:19.750075 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750067    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:16:19.750075 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750077    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret podName:ff5e36dc-a802-4ffd-99ec-db7ceb2429ba nodeName:}" failed. No retries permitted until 2026-07-01 17:16:27.750059199 +0000 UTC m=+18.155719955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret") pod "global-pull-secret-syncer-psbvf" (UID: "ff5e36dc-a802-4ffd-99ec-db7ceb2429ba") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:19.750223 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750081    2576 projected.go:196] Error preparing data for projected volume kube-api-access-86qzs for pod openshift-network-diagnostics/network-check-target-nqm5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:19.750223 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750134    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs podName:a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:27.750117782 +0000 UTC m=+18.155778540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-86qzs" (UniqueName: "kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs") pod "network-check-target-nqm5t" (UID: "a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:19.750223 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750178    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:19.750223 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:19.750211    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:27.75019964 +0000 UTC m=+18.155860397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:20.044179 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:20.043564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:20.044179 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:20.043672    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:20.044179 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:20.043787    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:20.044179 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:20.043892    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:21.043154 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:21.043122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:21.043357 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:21.043259    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:22.043670 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:22.043637    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:22.044123 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:22.043770    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:22.044123 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:22.043637    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:22.044123 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:22.043874    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:23.042874 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:23.042836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:23.043052 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:23.042968    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:24.043357 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:24.043313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:24.043813 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:24.043313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:24.043813 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:24.043461    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:24.043813 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:24.043545    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:25.043143 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:25.043097    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:25.043341 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:25.043246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:26.043265 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:26.043228    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:26.043265 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:26.043247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:26.043771 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:26.043371    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:26.043771 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:26.043498    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:27.043326 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:27.043296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:27.043784 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.043402    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:27.808642 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:27.808605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:27.808839 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:27.808662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:27.808839 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:27.808716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:27.808839 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808800    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:27.808839 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808827    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:27.809092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808848    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:16:27.809092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808864    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:16:27.809092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808880    2576 projected.go:196] Error preparing data for projected volume kube-api-access-86qzs for pod openshift-network-diagnostics/network-check-target-nqm5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:27.809092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808888    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret podName:ff5e36dc-a802-4ffd-99ec-db7ceb2429ba nodeName:}" failed. No retries permitted until 2026-07-01 17:16:43.808864531 +0000 UTC m=+34.214525290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret") pod "global-pull-secret-syncer-psbvf" (UID: "ff5e36dc-a802-4ffd-99ec-db7ceb2429ba") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:27.809092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808911    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:43.808900007 +0000 UTC m=+34.214560768 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:27.809092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:27.808928    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs podName:a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:43.808919828 +0000 UTC m=+34.214580600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-86qzs" (UniqueName: "kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs") pod "network-check-target-nqm5t" (UID: "a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:28.043261 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:28.043225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:28.043430 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:28.043366    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:28.043430 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:28.043415    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:28.043879 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:28.043503    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:29.042743 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:29.042701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:29.042924 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:29.042839    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:30.044220 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:30.044186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:30.044728 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:30.044306    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:30.044728 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:30.044301    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:30.044728 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:30.044412    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:31.043166 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:31.043132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:31.043339 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:31.043252    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:32.044143 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.043657    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:32.044143 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:32.043784    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:32.044874 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.044723    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:32.044874 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:32.044834    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:32.139291 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.139078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" event={"ID":"a2c3d4ff9542796e61a3b334a5839879","Type":"ContainerStarted","Data":"01bbbb7e9446b15a99ef88df3a001caa2fb5eb6537c6ece3d43897196380d5be"}
Jul 01 17:16:32.141640 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.141554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5v4vn" event={"ID":"5a686c1c-ce50-4e13-82d7-2296f4797d30","Type":"ContainerStarted","Data":"a3dcbf9e513e18487d6a7979a9b2e6c2867b208e7619347426311c23daff006a"}
Jul 01 17:16:32.143845 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.143758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vczhz" event={"ID":"f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9","Type":"ContainerStarted","Data":"0834b9f9ff91078150aed253e88dcafc6b28ac4f9eb12c84d73b4327853a6920"}
Jul 01 17:16:32.153160 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.152902    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-95.ec2.internal" podStartSLOduration=21.152882688 podStartE2EDuration="21.152882688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:32.15274789 +0000 UTC m=+22.558408667" watchObservedRunningTime="2026-07-01 17:16:32.152882688 +0000 UTC m=+22.558543464"
Jul 01 17:16:32.171915 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.171875    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5v4vn" podStartSLOduration=2.850254465 podStartE2EDuration="22.171859065s" totalImagesPullingTime="19.3216046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:32.171283623 +0000 UTC m=+22.576944399" watchObservedRunningTime="2026-07-01 17:16:32.171859065 +0000 UTC m=+22.577519840"
Jul 01 17:16:32.188032 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:32.187990    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vczhz" podStartSLOduration=2.810225385 podStartE2EDuration="22.187973832s" totalImagesPullingTime="19.377748447s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:32.187638467 +0000 UTC m=+22.593299245" watchObservedRunningTime="2026-07-01 17:16:32.187973832 +0000 UTC m=+22.593634607"
Jul 01 17:16:33.043217 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.043038    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:33.043409 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:33.043306    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:33.148958 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.148920    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp" event={"ID":"f720c753-1293-4ea0-ac00-7b4ca86b573e","Type":"ContainerStarted","Data":"14cbd4ace577884b8986d4d04638e5cf4d1b0801a06e887d0e57c2814d919765"}
Jul 01 17:16:33.150587 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.150554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9hwb" event={"ID":"e17c5967-c89a-47b9-9d37-3281f0f2b99c","Type":"ContainerStarted","Data":"74c11000cf1d0cf527b5fe21475f2cccdaa6a349aeec02c650d1bdf17cee7ec4"}
Jul 01 17:16:33.152107 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.152074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jcctl" event={"ID":"7373fb09-974e-47d2-8064-e91523f0f494","Type":"ContainerStarted","Data":"a3bffcc0281ec3f54db473565aa3e6e774215239d571e3db5eebcdf3b096afb2"}
Jul 01 17:16:33.155009 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.154991    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:16:33.155341 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.155321    2576 generic.go:356] "Generic (PLEG): container finished" podID="c12fa6c4-e35f-4aa2-8b8e-f274469878a9" containerID="a95243c0446489f130284cc8f3fb6fd2e998df02557130227f39363e851df66b" exitCode=1
Jul 01 17:16:33.155425 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.155398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"835a2f2470058fe89e46e953c3fb5223186e4d91be872b54af712ee0c573c867"}
Jul 01 17:16:33.155489 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.155474    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"0a74c1a36ec06ca1795d200349f186d99c985dd8cd46a88bfc5cd1807a373362"}
Jul 01 17:16:33.155562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.155546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"abf445030320eac41e61d83123b347d47aede624ba10453202f9aa1ca339df3c"}
Jul 01 17:16:33.155611 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.155595    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"be0cbe08d0daf7e6713ac9b7f80b2df6b187de8650780e6b37b342e04c5820ff"}
Jul 01 17:16:33.155659 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.155643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerDied","Data":"a95243c0446489f130284cc8f3fb6fd2e998df02557130227f39363e851df66b"}
Jul 01 17:16:33.155737 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.155723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"b6d000bab8f0458a591d7933db274a514ccc2009ff5a576e21c69e758f577bd3"}
Jul 01 17:16:33.156986 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.156955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lm9pd" event={"ID":"c0ba3c39-1752-4fac-b0d3-8868786e2412","Type":"ContainerStarted","Data":"8e5a3ff94ea90fcaa40868b063d90c50071941b6beaaecccc0876cf722c1d135"}
Jul 01 17:16:33.158312 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.158282    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88pqd" event={"ID":"97908b6c-1862-4784-bb3a-463d7a4cc872","Type":"ContainerStarted","Data":"25d69cd5916d061f4c374a8677845f9f52312bd61b11d14ea9ad50c8e1e0f9cf"}
Jul 01 17:16:33.159795 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.159766    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e3f55f5-2715-4960-8d8f-546f4a38d95b" containerID="c1183bd574c7e9326fd9e25d775be20a0e5ab2660c4bee0bf0ca6255ec245bbc" exitCode=0
Jul 01 17:16:33.159875 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.159843    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerDied","Data":"c1183bd574c7e9326fd9e25d775be20a0e5ab2660c4bee0bf0ca6255ec245bbc"}
Jul 01 17:16:33.167672 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.167636    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l9hwb" podStartSLOduration=3.862714698 podStartE2EDuration="23.167624811s" totalImagesPullingTime="19.304910113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:33.166912961 +0000 UTC m=+23.572573737" watchObservedRunningTime="2026-07-01 17:16:33.167624811 +0000 UTC m=+23.573285587"
Jul 01 17:16:33.182206 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.182165    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jcctl" podStartSLOduration=3.5390941160000002 podStartE2EDuration="23.182148446s" totalImagesPullingTime="19.64305433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:33.182061303 +0000 UTC m=+23.587722082" watchObservedRunningTime="2026-07-01 17:16:33.182148446 +0000 UTC m=+23.587809224"
Jul 01 17:16:33.214733 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.214636    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-88pqd" podStartSLOduration=3.907724496 podStartE2EDuration="23.214619016s" totalImagesPullingTime="19.30689452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:33.197951935 +0000 UTC m=+23.603612710" watchObservedRunningTime="2026-07-01 17:16:33.214619016 +0000 UTC m=+23.620279793"
Jul 01 17:16:33.235613 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.235558    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lm9pd" podStartSLOduration=3.924325177 podStartE2EDuration="23.235536336s" totalImagesPullingTime="19.311211159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:33.21414769 +0000 UTC m=+23.619808466" watchObservedRunningTime="2026-07-01 17:16:33.235536336 +0000 UTC m=+23.641197114"
Jul 01 17:16:33.695660 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:33.695628    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 01 17:16:34.046095 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:34.046065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:34.046317 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:34.046164    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:34.046317 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:34.046213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:34.046317 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:34.046296    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:34.053104 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:34.052992    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-01T17:16:33.695653096Z","UUID":"f495254f-1074-46df-9b40-206a23291f65","Handler":null,"Name":"","Endpoint":""}
Jul 01 17:16:34.054328 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:34.054310    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 01 17:16:34.054437 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:34.054342    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 01 17:16:34.164037 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:34.163976    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp" event={"ID":"f720c753-1293-4ea0-ac00-7b4ca86b573e","Type":"ContainerStarted","Data":"2a41e21361e6778b3404a65d3da1ef5e9d46da6773a3a09809b29281c98156e6"}
Jul 01 17:16:35.043584 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:35.043343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:35.043771 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:35.043644    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:35.168197 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:35.168154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp" event={"ID":"f720c753-1293-4ea0-ac00-7b4ca86b573e","Type":"ContainerStarted","Data":"1c1d407253900a24c50b88a6b1bba0b9bb545a732ae9040888a47d940e5cd1b3"}
Jul 01 17:16:35.170889 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:35.170870    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:16:35.171173 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:35.171155    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"320d4d9ceb1388af15f052c4c48a5d474308a35dd80ee70095488363cdb00be9"}
Jul 01 17:16:35.187798 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:35.187752    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6vsp" podStartSLOduration=7.623522422 podStartE2EDuration="25.187737955s" totalImagesPullingTime="17.564215533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:35.187447889 +0000 UTC m=+25.593108665" watchObservedRunningTime="2026-07-01 17:16:35.187737955 +0000 UTC m=+25.593398731"
Jul 01 17:16:35.994590 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:35.994563    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:36.043430 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:36.043392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:36.043583 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:36.043437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:36.043583 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:36.043543    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:36.043677 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:36.043643    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:37.042768 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.042737    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:37.043451 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:37.042867    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:37.179695 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.179669    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:16:37.180017 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.179996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"dd0be2ee13166aed6470b4d3251b2d4e1c038088ffbac4ccefd884009f6e1351"}
Jul 01 17:16:37.180374 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.180344    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:37.180507 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.180422    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:37.180507 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.180477    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:37.180507 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.180498    2576 scope.go:117] "RemoveContainer" containerID="a95243c0446489f130284cc8f3fb6fd2e998df02557130227f39363e851df66b"
Jul 01 17:16:37.197001 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.196006    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:37.198435 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.198379    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:16:37.334544 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.334426    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:37.335116 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:37.335092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:38.042655 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:38.042625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:38.042840 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:38.042625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:38.042840 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:38.042749    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:38.043349 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:38.042849    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:38.185286 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:38.185256    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:16:38.185637 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:38.185600    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" event={"ID":"c12fa6c4-e35f-4aa2-8b8e-f274469878a9","Type":"ContainerStarted","Data":"3bedc9285b46875d3d296ced915c696233ad7d3a3bd3b93100a8eac761c60691"}
Jul 01 17:16:38.186364 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:38.186349    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-88pqd"
Jul 01 17:16:38.215888 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:38.215850    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn" podStartSLOduration=8.483047286 podStartE2EDuration="28.215835664s" totalImagesPullingTime="19.732788378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:38.214474988 +0000 UTC m=+28.620135764" watchObservedRunningTime="2026-07-01 17:16:38.215835664 +0000 UTC m=+28.621496439"
Jul 01 17:16:39.043643 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:39.043617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:39.044028 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:39.043721    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:39.335307 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:39.335224    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psbvf"]
Jul 01 17:16:39.335443 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:39.335343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:39.335504 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:39.335444    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:39.335986 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:39.335965    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nqm5t"]
Jul 01 17:16:39.336099 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:39.336049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:39.336171 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:39.336144    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:39.337030 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:39.337004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tb5ht"]
Jul 01 17:16:39.337136 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:39.337099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:39.337210 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:39.337189    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:41.043051 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:41.043020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:41.043468 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:41.043020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:41.043468 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:41.043135    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:41.043468 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:41.043245    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:41.043468 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:41.043027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:41.043468 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:41.043357    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:42.197342 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:42.197173    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e3f55f5-2715-4960-8d8f-546f4a38d95b" containerID="2bb02f5e33463bb3eab33f9ead150d410dfcd0c3e12db8220c4cd3cab9ac3cdc" exitCode=0
Jul 01 17:16:42.198010 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:42.197272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerDied","Data":"2bb02f5e33463bb3eab33f9ead150d410dfcd0c3e12db8220c4cd3cab9ac3cdc"}
Jul 01 17:16:43.043579 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:43.043549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:43.043741 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:43.043664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:43.043741 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.043672    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tb5ht" podUID="c7ba0dbe-db80-4c86-afc4-545af38d30d0"
Jul 01 17:16:43.043741 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:43.043703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:43.043849 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.043812    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-psbvf" podUID="ff5e36dc-a802-4ffd-99ec-db7ceb2429ba"
Jul 01 17:16:43.043929 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.043909    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nqm5t" podUID="a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59"
Jul 01 17:16:43.828886 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:43.828851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:43.828886 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:43.828903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:43.828940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829020    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829040    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829049    2576 projected.go:196] Error preparing data for projected volume kube-api-access-86qzs for pod openshift-network-diagnostics/network-check-target-nqm5t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829058    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829092    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs podName:a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:15.829079717 +0000 UTC m=+66.234740474 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-86qzs" (UniqueName: "kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs") pod "network-check-target-nqm5t" (UID: "a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829107    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret podName:ff5e36dc-a802-4ffd-99ec-db7ceb2429ba nodeName:}" failed. No retries permitted until 2026-07-01 17:17:15.829101202 +0000 UTC m=+66.234761960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret") pod "global-pull-secret-syncer-psbvf" (UID: "ff5e36dc-a802-4ffd-99ec-db7ceb2429ba") : object "kube-system"/"original-pull-secret" not registered
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829061    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:43.829417 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:43.829133    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:15.829126981 +0000 UTC m=+66.234787735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 01 17:16:44.204141 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.204112    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e3f55f5-2715-4960-8d8f-546f4a38d95b" containerID="a6b2b8eb267447a9b628f7bf02a033e01c9226e0b6497cf5852197ee38ba4e74" exitCode=0
Jul 01 17:16:44.204286 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.204160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerDied","Data":"a6b2b8eb267447a9b628f7bf02a033e01c9226e0b6497cf5852197ee38ba4e74"}
Jul 01 17:16:44.924690 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.924619    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-95.ec2.internal" event="NodeReady"
Jul 01 17:16:44.925051 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.924751    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 01 17:16:44.965844 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.965804    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-768d4f548c-ftpw7"]
Jul 01 17:16:44.967731 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.967715    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:44.970278 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.970245    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 01 17:16:44.970389 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.970279    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ggrg5\""
Jul 01 17:16:44.970389 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.970343    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 01 17:16:44.973959 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.970597    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 01 17:16:44.977751 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.977617    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 01 17:16:44.981030 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.981012    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kgg9j"]
Jul 01 17:16:44.982806 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.982792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:44.985161 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.985138    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-768d4f548c-ftpw7"]
Jul 01 17:16:44.985470 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.985453    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 01 17:16:44.985568 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.985549    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n25z7\""
Jul 01 17:16:44.985882 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.985859    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nvbqg"]
Jul 01 17:16:44.987360 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.987344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:44.988144 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.988126    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 01 17:16:44.990173 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.990157    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 01 17:16:44.990250 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.990176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 01 17:16:44.990415 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.990402    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 01 17:16:44.992274 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:44.992259    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xqdpm\""
Jul 01 17:16:45.003035 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.003008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kgg9j"]
Jul 01 17:16:45.003562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.003539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvbqg"]
Jul 01 17:16:45.043169 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.043137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:16:45.043319 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.043182    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:16:45.043319 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.043234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:16:45.046617 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.046589    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gs6qb\""
Jul 01 17:16:45.046865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.046847    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 17:16:45.046961 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.046893    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hbjh4\""
Jul 01 17:16:45.046961 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.046951    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 17:16:45.047060 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.047000    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 17:16:45.047270 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.047258    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 17:16:45.139074 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.139235 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-installation-pull-secrets\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139235 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bce2dd3-45c1-4fbe-b853-3c97b6508100-tmp-dir\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.139235 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppns\" (UniqueName: \"kubernetes.io/projected/18279d9a-bdc9-48f6-bb38-dfe5711b3460-kube-api-access-2ppns\") pod \"ingress-canary-nvbqg\" (UID: \"18279d9a-bdc9-48f6-bb38-dfe5711b3460\") " pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:45.139235 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-bound-sa-token\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139423 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-image-registry-private-configuration\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139423 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18279d9a-bdc9-48f6-bb38-dfe5711b3460-cert\") pod \"ingress-canary-nvbqg\" (UID: \"18279d9a-bdc9-48f6-bb38-dfe5711b3460\") " pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:45.139423 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f793d210-92d9-4705-882c-064c9fdf2c23-ca-trust-extracted\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139423 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-registry-certificates\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139423 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjh22\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-kube-api-access-gjh22\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139685 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpc22\" (UniqueName: \"kubernetes.io/projected/6bce2dd3-45c1-4fbe-b853-3c97b6508100-kube-api-access-mpc22\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.139685 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139685 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-trusted-ca\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.139685 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.139579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce2dd3-45c1-4fbe-b853-3c97b6508100-config-volume\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.149360 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.149331    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:45.238901 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.238869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:45.240729 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.240710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bce2dd3-45c1-4fbe-b853-3c97b6508100-tmp-dir\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.240804 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.240792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppns\" (UniqueName: \"kubernetes.io/projected/18279d9a-bdc9-48f6-bb38-dfe5711b3460-kube-api-access-2ppns\") pod \"ingress-canary-nvbqg\" (UID: \"18279d9a-bdc9-48f6-bb38-dfe5711b3460\") " pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:45.240846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.240824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-bound-sa-token\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.240898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.240845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-image-registry-private-configuration\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.240949 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.240932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18279d9a-bdc9-48f6-bb38-dfe5711b3460-cert\") pod \"ingress-canary-nvbqg\" (UID: \"18279d9a-bdc9-48f6-bb38-dfe5711b3460\") " pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:45.240996 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.240973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f793d210-92d9-4705-882c-064c9fdf2c23-ca-trust-extracted\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241044 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-registry-certificates\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjh22\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-kube-api-access-gjh22\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241045    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bce2dd3-45c1-4fbe-b853-3c97b6508100-tmp-dir\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.241233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpc22\" (UniqueName: \"kubernetes.io/projected/6bce2dd3-45c1-4fbe-b853-3c97b6508100-kube-api-access-mpc22\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.241311 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241367 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-trusted-ca\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241416 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce2dd3-45c1-4fbe-b853-3c97b6508100-config-volume\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.241416 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241384    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f793d210-92d9-4705-882c-064c9fdf2c23-ca-trust-extracted\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241416 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.241395    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:45.241416 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.241411    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768d4f548c-ftpw7: secret "image-registry-tls" not found
Jul 01 17:16:45.241604 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.241462    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls podName:f793d210-92d9-4705-882c-064c9fdf2c23 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:45.741444184 +0000 UTC m=+36.147104956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls") pod "image-registry-768d4f548c-ftpw7" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23") : secret "image-registry-tls" not found
Jul 01 17:16:45.241604 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.241468    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:45.241604 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.241568    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls podName:6bce2dd3-45c1-4fbe-b853-3c97b6508100 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:45.741552416 +0000 UTC m=+36.147213177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls") pod "dns-default-kgg9j" (UID: "6bce2dd3-45c1-4fbe-b853-3c97b6508100") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:45.241749 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241637    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-registry-certificates\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241749 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.241749 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-installation-pull-secrets\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.241885 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.241848    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6bce2dd3-45c1-4fbe-b853-3c97b6508100-config-volume\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.242401 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.242380    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-trusted-ca\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.245121 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.245098    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-image-registry-private-configuration\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.245211 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.245164    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-installation-pull-secrets\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.245211 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.245188    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18279d9a-bdc9-48f6-bb38-dfe5711b3460-cert\") pod \"ingress-canary-nvbqg\" (UID: \"18279d9a-bdc9-48f6-bb38-dfe5711b3460\") " pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:45.252238 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.252212    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpc22\" (UniqueName: \"kubernetes.io/projected/6bce2dd3-45c1-4fbe-b853-3c97b6508100-kube-api-access-mpc22\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.252674 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.252651    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppns\" (UniqueName: \"kubernetes.io/projected/18279d9a-bdc9-48f6-bb38-dfe5711b3460-kube-api-access-2ppns\") pod \"ingress-canary-nvbqg\" (UID: \"18279d9a-bdc9-48f6-bb38-dfe5711b3460\") " pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:45.252794 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.252779    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjh22\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-kube-api-access-gjh22\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.253995 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.253980    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-bound-sa-token\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.297280 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.297250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvbqg"
Jul 01 17:16:45.436787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.436736    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvbqg"]
Jul 01 17:16:45.448608 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:45.448579    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18279d9a_bdc9_48f6_bb38_dfe5711b3460.slice/crio-9f00447ef2359d07f55ee03a5132befc5b638a80e015e0bfed2172a688c49501 WatchSource:0}: Error finding container 9f00447ef2359d07f55ee03a5132befc5b638a80e015e0bfed2172a688c49501: Status 404 returned error can't find the container with id 9f00447ef2359d07f55ee03a5132befc5b638a80e015e0bfed2172a688c49501
Jul 01 17:16:45.482500 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.482460    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:45.584432 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.584318    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:45.745267 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.745222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:45.745404 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.745370    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:45.745448 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:45.745427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:45.745480 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.745443    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls podName:6bce2dd3-45c1-4fbe-b853-3c97b6508100 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:46.745424497 +0000 UTC m=+37.151085253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls") pod "dns-default-kgg9j" (UID: "6bce2dd3-45c1-4fbe-b853-3c97b6508100") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:45.745567 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.745510    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:45.745567 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.745539    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768d4f548c-ftpw7: secret "image-registry-tls" not found
Jul 01 17:16:45.745635 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:45.745578    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls podName:f793d210-92d9-4705-882c-064c9fdf2c23 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:46.745565593 +0000 UTC m=+37.151226359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls") pod "image-registry-768d4f548c-ftpw7" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23") : secret "image-registry-tls" not found
Jul 01 17:16:46.212797 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:46.212761    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e3f55f5-2715-4960-8d8f-546f4a38d95b" containerID="e46ac6aed4f57407a464c295faff843000f144f9d11c6724c89780b44a8f384c" exitCode=0
Jul 01 17:16:46.213315 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:46.212849    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerDied","Data":"e46ac6aed4f57407a464c295faff843000f144f9d11c6724c89780b44a8f384c"}
Jul 01 17:16:46.214486 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:46.214259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvbqg" event={"ID":"18279d9a-bdc9-48f6-bb38-dfe5711b3460","Type":"ContainerStarted","Data":"9f00447ef2359d07f55ee03a5132befc5b638a80e015e0bfed2172a688c49501"}
Jul 01 17:16:46.755799 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:46.755761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:46.755984 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:46.755883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:46.755984 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:46.755936    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:46.756091 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:46.756018    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:46.756091 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:46.756034    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768d4f548c-ftpw7: secret "image-registry-tls" not found
Jul 01 17:16:46.756091 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:46.756019    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls podName:6bce2dd3-45c1-4fbe-b853-3c97b6508100 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:48.755997916 +0000 UTC m=+39.161658687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls") pod "dns-default-kgg9j" (UID: "6bce2dd3-45c1-4fbe-b853-3c97b6508100") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:46.756091 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:46.756092    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls podName:f793d210-92d9-4705-882c-064c9fdf2c23 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:48.756078028 +0000 UTC m=+39.161738785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls") pod "image-registry-768d4f548c-ftpw7" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23") : secret "image-registry-tls" not found
Jul 01 17:16:48.777259 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:48.777067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:48.777911 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:48.777269    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:48.777911 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:48.777291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:48.777911 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:48.777295    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768d4f548c-ftpw7: secret "image-registry-tls" not found
Jul 01 17:16:48.777911 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:48.777347    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:48.777911 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:48.777401    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls podName:6bce2dd3-45c1-4fbe-b853-3c97b6508100 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:52.777381172 +0000 UTC m=+43.183041926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls") pod "dns-default-kgg9j" (UID: "6bce2dd3-45c1-4fbe-b853-3c97b6508100") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:48.777911 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:48.777422    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls podName:f793d210-92d9-4705-882c-064c9fdf2c23 nodeName:}" failed. No retries permitted until 2026-07-01 17:16:52.777412357 +0000 UTC m=+43.183073121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls") pod "image-registry-768d4f548c-ftpw7" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23") : secret "image-registry-tls" not found
Jul 01 17:16:49.222602 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:49.222556    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvbqg" event={"ID":"18279d9a-bdc9-48f6-bb38-dfe5711b3460","Type":"ContainerStarted","Data":"75dc13dea4ccda3bf49172ab719564c23faa369db0ffd0b98c077921c8348ec4"}
Jul 01 17:16:49.240858 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:49.240791    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nvbqg" podStartSLOduration=2.474649594 podStartE2EDuration="5.240766253s" totalImagesPullingTime="2.766116659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:49.24065076 +0000 UTC m=+39.646311552" watchObservedRunningTime="2026-07-01 17:16:49.240766253 +0000 UTC m=+39.646427029"
Jul 01 17:16:49.606682 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:49.606588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:49.855552 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:49.855499    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:49.933457 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:49.933359    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:50.452363 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:50.452308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:50.520153 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:50.520119    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:51.229125 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:51.229097    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e3f55f5-2715-4960-8d8f-546f4a38d95b" containerID="50deb439088f2261c59da6ba789a12e44645d493aaa995d8e325db174fa0ccf5" exitCode=0
Jul 01 17:16:51.229497 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:51.229146    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerDied","Data":"50deb439088f2261c59da6ba789a12e44645d493aaa995d8e325db174fa0ccf5"}
Jul 01 17:16:51.709445 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:51.709410    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log" limit=1024
Jul 01 17:16:52.235045 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:52.235015    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e3f55f5-2715-4960-8d8f-546f4a38d95b" containerID="46c00683c6bc159558c4383fc16f3b1b6bd34ce9db745db1ca5e2112c1536fb4" exitCode=0
Jul 01 17:16:52.235405 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:52.235079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerDied","Data":"46c00683c6bc159558c4383fc16f3b1b6bd34ce9db745db1ca5e2112c1536fb4"}
Jul 01 17:16:52.806392 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:52.806358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:16:52.806578 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:52.806412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:16:52.806578 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:52.806510    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:16:52.806578 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:52.806536    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:16:52.806578 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:52.806544    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768d4f548c-ftpw7: secret "image-registry-tls" not found
Jul 01 17:16:52.806713 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:52.806594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls podName:6bce2dd3-45c1-4fbe-b853-3c97b6508100 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:00.806578485 +0000 UTC m=+51.212239240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls") pod "dns-default-kgg9j" (UID: "6bce2dd3-45c1-4fbe-b853-3c97b6508100") : secret "dns-default-metrics-tls" not found
Jul 01 17:16:52.806713 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:16:52.806608    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls podName:f793d210-92d9-4705-882c-064c9fdf2c23 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:00.806602161 +0000 UTC m=+51.212262914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls") pod "image-registry-768d4f548c-ftpw7" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23") : secret "image-registry-tls" not found
Jul 01 17:16:53.240333 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.240301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" event={"ID":"7e3f55f5-2715-4960-8d8f-546f4a38d95b","Type":"ContainerStarted","Data":"1e2b218605c93d18f083ec3f5d79fdf1d710be071f7c2c3df0f1dd9b36ed48b1"}
Jul 01 17:16:53.267514 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.267400    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nh6jx" podStartSLOduration=9.203774806 podStartE2EDuration="43.267380464s" totalImagesPullingTime="34.063605658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:16:53.265586523 +0000 UTC m=+43.671247299" watchObservedRunningTime="2026-07-01 17:16:53.267380464 +0000 UTC m=+43.673041218"
Jul 01 17:16:53.972148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.972115    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"]
Jul 01 17:16:53.987926 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.987899    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:53.991365 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.991340    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 01 17:16:53.991365 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.991359    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 01 17:16:53.991573 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.991417    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"]
Jul 01 17:16:53.992720 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.992695    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 01 17:16:53.992720 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.992726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 01 17:16:53.992898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.992761    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 01 17:16:53.993022 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.993008    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 01 17:16:53.993181 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:53.993169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 01 17:16:54.012605 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.012579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-ca\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.012705 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.012652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.012705 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.012672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-hub\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.012806 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.012707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c759749d-5561-4f8b-9874-1c65130746ad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.012806 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.012731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.012806 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.012760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mq5n\" (UniqueName: \"kubernetes.io/projected/c759749d-5561-4f8b-9874-1c65130746ad-kube-api-access-9mq5n\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.113317 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.113285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.113317 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.113317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-hub\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.113558 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.113352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c759749d-5561-4f8b-9874-1c65130746ad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.113558 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.113377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.113558 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.113396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mq5n\" (UniqueName: \"kubernetes.io/projected/c759749d-5561-4f8b-9874-1c65130746ad-kube-api-access-9mq5n\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.113558 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.113417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-ca\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.114083 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.114061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c759749d-5561-4f8b-9874-1c65130746ad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.116939 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.116910    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-ca\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.116939 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.116926    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.117287 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.117271    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-hub\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.117439 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.117420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c759749d-5561-4f8b-9874-1c65130746ad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.122766 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.122743    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mq5n\" (UniqueName: \"kubernetes.io/projected/c759749d-5561-4f8b-9874-1c65130746ad-kube-api-access-9mq5n\") pod \"cluster-proxy-proxy-agent-fb7994cf9-bqbqk\" (UID: \"c759749d-5561-4f8b-9874-1c65130746ad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.310383 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.310305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:16:54.459832 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:54.459798    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"]
Jul 01 17:16:54.461705 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:16:54.461678    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc759749d_5561_4f8b_9874_1c65130746ad.slice/crio-76869ef550f8cfe2c6ba61a964033adefff0789cf8ee71aa5e832edd815e85d0 WatchSource:0}: Error finding container 76869ef550f8cfe2c6ba61a964033adefff0789cf8ee71aa5e832edd815e85d0: Status 404 returned error can't find the container with id 76869ef550f8cfe2c6ba61a964033adefff0789cf8ee71aa5e832edd815e85d0
Jul 01 17:16:55.245820 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:55.245785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" event={"ID":"c759749d-5561-4f8b-9874-1c65130746ad","Type":"ContainerStarted","Data":"76869ef550f8cfe2c6ba61a964033adefff0789cf8ee71aa5e832edd815e85d0"}
Jul 01 17:16:59.257195 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:16:59.257169    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" event={"ID":"c759749d-5561-4f8b-9874-1c65130746ad","Type":"ContainerStarted","Data":"01068549f1a1901d7ed92506194c6100206a56803106d310e49f88ee28c9cc07"}
Jul 01 17:17:00.261787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:00.261754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" event={"ID":"c759749d-5561-4f8b-9874-1c65130746ad","Type":"ContainerStarted","Data":"ec11c0b750a500739a39cafadb30671ee85e3d949986b3ecd837860b9ea9e08e"}
Jul 01 17:17:00.261787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:00.261795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" event={"ID":"c759749d-5561-4f8b-9874-1c65130746ad","Type":"ContainerStarted","Data":"4edef91fdf114062a6372d695d7dfbbd274c136f2ead3d83aaf4f1cc5ba4dc65"}
Jul 01 17:17:00.281989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:00.281944    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" podStartSLOduration=2.602565214 podStartE2EDuration="7.281931305s" totalImagesPullingTime="4.679366091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:17:00.281732522 +0000 UTC m=+50.687393298" watchObservedRunningTime="2026-07-01 17:17:00.281931305 +0000 UTC m=+50.687592072"
Jul 01 17:17:00.874062 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:00.874016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:17:00.874231 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:00.874078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:17:00.874231 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:00.874172    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:17:00.874231 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:00.874199    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768d4f548c-ftpw7: secret "image-registry-tls" not found
Jul 01 17:17:00.874231 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:00.874175    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:17:00.874384 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:00.874253    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls podName:6bce2dd3-45c1-4fbe-b853-3c97b6508100 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:16.874240277 +0000 UTC m=+67.279901031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls") pod "dns-default-kgg9j" (UID: "6bce2dd3-45c1-4fbe-b853-3c97b6508100") : secret "dns-default-metrics-tls" not found
Jul 01 17:17:00.874384 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:00.874266    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls podName:f793d210-92d9-4705-882c-064c9fdf2c23 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:16.874260325 +0000 UTC m=+67.279921078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls") pod "image-registry-768d4f548c-ftpw7" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23") : secret "image-registry-tls" not found
Jul 01 17:17:09.198549 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:09.198486    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtzvn"
Jul 01 17:17:15.887361 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.887315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:17:15.887361 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.887370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:17:15.887976 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.887414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:17:15.890397 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.890373    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 01 17:17:15.890397 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.890376    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 01 17:17:15.890573 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.890406    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 01 17:17:15.897945 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:15.897923    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 01 17:17:15.898032 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:15.898019    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs podName:c7ba0dbe-db80-4c86-afc4-545af38d30d0 nodeName:}" failed. No retries permitted until 2026-07-01 17:18:19.89799096 +0000 UTC m=+130.303651717 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs") pod "network-metrics-daemon-tb5ht" (UID: "c7ba0dbe-db80-4c86-afc4-545af38d30d0") : secret "metrics-daemon-secret" not found
Jul 01 17:17:15.900133 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.900109    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 01 17:17:15.900258 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.900145    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff5e36dc-a802-4ffd-99ec-db7ceb2429ba-original-pull-secret\") pod \"global-pull-secret-syncer-psbvf\" (UID: \"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba\") " pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:17:15.910648 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.910619    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86qzs\" (UniqueName: \"kubernetes.io/projected/a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59-kube-api-access-86qzs\") pod \"network-check-target-nqm5t\" (UID: \"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59\") " pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:17:15.955692 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.955665    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hbjh4\""
Jul 01 17:17:15.963236 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.963206    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:17:15.963309 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:15.963211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-psbvf"
Jul 01 17:17:16.094390 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:16.094355    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nqm5t"]
Jul 01 17:17:16.098359 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:17:16.098333    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f9ca68_b7ed_4b9e_a028_62e8c7d19a59.slice/crio-4982c6e16dc79daff409f65d139d67bcf5455e45a16ddb822310b6d59c454007 WatchSource:0}: Error finding container 4982c6e16dc79daff409f65d139d67bcf5455e45a16ddb822310b6d59c454007: Status 404 returned error can't find the container with id 4982c6e16dc79daff409f65d139d67bcf5455e45a16ddb822310b6d59c454007
Jul 01 17:17:16.106676 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:16.106653    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-psbvf"]
Jul 01 17:17:16.109582 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:17:16.109558    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5e36dc_a802_4ffd_99ec_db7ceb2429ba.slice/crio-4fa0e08344676f4b1268539e732ee157185816c6d8c7878a230e21bd90436abd WatchSource:0}: Error finding container 4fa0e08344676f4b1268539e732ee157185816c6d8c7878a230e21bd90436abd: Status 404 returned error can't find the container with id 4fa0e08344676f4b1268539e732ee157185816c6d8c7878a230e21bd90436abd
Jul 01 17:17:16.300850 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:16.300813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psbvf" event={"ID":"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba","Type":"ContainerStarted","Data":"4fa0e08344676f4b1268539e732ee157185816c6d8c7878a230e21bd90436abd"}
Jul 01 17:17:16.301937 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:16.301913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nqm5t" event={"ID":"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59","Type":"ContainerStarted","Data":"4982c6e16dc79daff409f65d139d67bcf5455e45a16ddb822310b6d59c454007"}
Jul 01 17:17:16.895075 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:16.894766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:17:16.895075 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:16.894851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:17:16.895075 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:16.895001    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 01 17:17:16.895075 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:16.895026    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 01 17:17:16.895075 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:16.895039    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-768d4f548c-ftpw7: secret "image-registry-tls" not found
Jul 01 17:17:16.895703 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:16.895094    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls podName:6bce2dd3-45c1-4fbe-b853-3c97b6508100 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:48.89507181 +0000 UTC m=+99.300732567 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls") pod "dns-default-kgg9j" (UID: "6bce2dd3-45c1-4fbe-b853-3c97b6508100") : secret "dns-default-metrics-tls" not found
Jul 01 17:17:16.895703 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:16.895116    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls podName:f793d210-92d9-4705-882c-064c9fdf2c23 nodeName:}" failed. No retries permitted until 2026-07-01 17:17:48.89510651 +0000 UTC m=+99.300767269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls") pod "image-registry-768d4f548c-ftpw7" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23") : secret "image-registry-tls" not found
Jul 01 17:17:20.314787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:20.314748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nqm5t" event={"ID":"a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59","Type":"ContainerStarted","Data":"613e7c4ef907caab518f270ed2bbb886ae061958755092e9f5c018ace342f052"}
Jul 01 17:17:20.315269 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:20.314883    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:17:20.332120 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:20.332066    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nqm5t" podStartSLOduration=67.082926156 podStartE2EDuration="1m10.332050884s" totalImagesPullingTime="3.249124728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:17:20.330751082 +0000 UTC m=+70.736411870" watchObservedRunningTime="2026-07-01 17:17:20.332050884 +0000 UTC m=+70.737711659"
Jul 01 17:17:21.319656 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:21.319566    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-psbvf" event={"ID":"ff5e36dc-a802-4ffd-99ec-db7ceb2429ba","Type":"ContainerStarted","Data":"dfd6a5fd143e11de88d6b440852989a1bdbf193a6217192992a73f29fba2d15d"}
Jul 01 17:17:21.338015 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:21.336050    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-psbvf" podStartSLOduration=66.478018503 podStartE2EDuration="1m11.336031083s" totalImagesPullingTime="4.85801258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:17:21.334460325 +0000 UTC m=+71.740121091" watchObservedRunningTime="2026-07-01 17:17:21.336031083 +0000 UTC m=+71.741691860"
Jul 01 17:17:37.363620 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.363483    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vhkql"]
Jul 01 17:17:37.366429 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.366409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.368892 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.368861    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 01 17:17:37.368892 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.368884    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 01 17:17:37.369143 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.369122    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 01 17:17:37.369191 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.369165    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 01 17:17:37.370277 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.370262    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5vpc8\""
Jul 01 17:17:37.382270 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.382246    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhkql"]
Jul 01 17:17:37.455665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.455634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-data-volume\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.455816 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.455670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7j5r\" (UniqueName: \"kubernetes.io/projected/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-api-access-s7j5r\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.455816 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.455700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.455816 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.455803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-crio-socket\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.455925 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.455821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.556385 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.556356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-crio-socket\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.556385 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.556388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.556534 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.556406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-data-volume\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.556534 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.556423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7j5r\" (UniqueName: \"kubernetes.io/projected/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-api-access-s7j5r\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.556534 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.556458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.556534 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:37.556492    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:17:37.556673 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.556492    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-crio-socket\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.556673 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:37.556594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:38.056574033 +0000 UTC m=+88.462234788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : configmap "kube-rbac-proxy" not found
Jul 01 17:17:37.556673 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:37.556627    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:37.556793 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:37.556683    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:38.056668126 +0000 UTC m=+88.462328894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:37.556793 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.556744    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-data-volume\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:37.565191 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:37.565170    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7j5r\" (UniqueName: \"kubernetes.io/projected/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-api-access-s7j5r\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:38.060331 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:38.060300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:38.060498 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:38.060350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:38.060498 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:38.060426    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:17:38.060498 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:38.060453    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:38.060498 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:38.060495    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:39.060477019 +0000 UTC m=+89.466137787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : configmap "kube-rbac-proxy" not found
Jul 01 17:17:38.060656 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:38.060513    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:39.060506731 +0000 UTC m=+89.466167485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:39.069313 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:39.069266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:39.069313 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:39.069323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:39.069726 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:39.069377    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:17:39.069726 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:39.069426    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:39.069726 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:39.069438    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:41.069423064 +0000 UTC m=+91.475083821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : configmap "kube-rbac-proxy" not found
Jul 01 17:17:39.069726 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:39.069494    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:41.069479586 +0000 UTC m=+91.475140356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:39.458789 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:39.458765    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lm9pd_c0ba3c39-1752-4fac-b0d3-8868786e2412/dns-node-resolver/0.log"
Jul 01 17:17:40.659141 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:40.659099    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jcctl_7373fb09-974e-47d2-8064-e91523f0f494/node-ca/0.log"
Jul 01 17:17:41.084430 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:41.084387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:41.084643 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:41.084440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:41.084643 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:41.084512    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:17:41.084643 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:41.084587    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:41.084643 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:41.084599    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:45.084583507 +0000 UTC m=+95.490244264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : configmap "kube-rbac-proxy" not found
Jul 01 17:17:41.084643 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:41.084639    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:45.08462863 +0000 UTC m=+95.490289384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:41.259140 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:41.259107    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nvbqg_18279d9a-bdc9-48f6-bb38-dfe5711b3460/serve-healthcheck-canary/0.log"
Jul 01 17:17:45.113702 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:45.113661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:45.114092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:45.113745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:45.114092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:45.113811    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 01 17:17:45.114092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:45.113834    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:45.114092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:45.113883    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:53.113869499 +0000 UTC m=+103.519530252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : configmap "kube-rbac-proxy" not found
Jul 01 17:17:45.114092 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:17:45.113897    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls podName:0cf27d80-b131-4905-8ec0-6a15ea46fc8e nodeName:}" failed. No retries permitted until 2026-07-01 17:17:53.113889784 +0000 UTC m=+103.519550537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhkql" (UID: "0cf27d80-b131-4905-8ec0-6a15ea46fc8e") : secret "insights-runtime-extractor-tls" not found
Jul 01 17:17:48.940833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:48.940768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:17:48.941240 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:48.940947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:17:48.943172 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:48.943143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bce2dd3-45c1-4fbe-b853-3c97b6508100-metrics-tls\") pod \"dns-default-kgg9j\" (UID: \"6bce2dd3-45c1-4fbe-b853-3c97b6508100\") " pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:17:48.943271 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:48.943154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"image-registry-768d4f548c-ftpw7\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") " pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:17:49.180847 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.180757    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ggrg5\""
Jul 01 17:17:49.188412 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.188391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:17:49.196234 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.196215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n25z7\""
Jul 01 17:17:49.203986 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.203965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:17:49.328513 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.328485    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-768d4f548c-ftpw7"]
Jul 01 17:17:49.331703 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:17:49.331679    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf793d210_92d9_4705_882c_064c9fdf2c23.slice/crio-939e139424229bc9dc16ededcf28e765d14b2b68c1f9b64f5b361bb86f9adea7 WatchSource:0}: Error finding container 939e139424229bc9dc16ededcf28e765d14b2b68c1f9b64f5b361bb86f9adea7: Status 404 returned error can't find the container with id 939e139424229bc9dc16ededcf28e765d14b2b68c1f9b64f5b361bb86f9adea7
Jul 01 17:17:49.345391 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.345359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kgg9j"]
Jul 01 17:17:49.347943 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:17:49.347919    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bce2dd3_45c1_4fbe_b853_3c97b6508100.slice/crio-d98e1631ca7aa9ebb1b1e90b02674b4037ca031dd44ba6771ff91a69a3342ee0 WatchSource:0}: Error finding container d98e1631ca7aa9ebb1b1e90b02674b4037ca031dd44ba6771ff91a69a3342ee0: Status 404 returned error can't find the container with id d98e1631ca7aa9ebb1b1e90b02674b4037ca031dd44ba6771ff91a69a3342ee0
Jul 01 17:17:49.401978 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.401945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kgg9j" event={"ID":"6bce2dd3-45c1-4fbe-b853-3c97b6508100","Type":"ContainerStarted","Data":"d98e1631ca7aa9ebb1b1e90b02674b4037ca031dd44ba6771ff91a69a3342ee0"}
Jul 01 17:17:49.403422 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.403398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7" event={"ID":"f793d210-92d9-4705-882c-064c9fdf2c23","Type":"ContainerStarted","Data":"d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666"}
Jul 01 17:17:49.403569 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.403436    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7" event={"ID":"f793d210-92d9-4705-882c-064c9fdf2c23","Type":"ContainerStarted","Data":"939e139424229bc9dc16ededcf28e765d14b2b68c1f9b64f5b361bb86f9adea7"}
Jul 01 17:17:49.403569 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.403544    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:17:49.424794 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:49.424733    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7" podStartSLOduration=99.424712571 podStartE2EDuration="1m39.424712571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:17:49.42455264 +0000 UTC m=+99.830213418" watchObservedRunningTime="2026-07-01 17:17:49.424712571 +0000 UTC m=+99.830373351"
Jul 01 17:17:51.322596 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:51.322562    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nqm5t"
Jul 01 17:17:52.416472 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:52.416439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kgg9j" event={"ID":"6bce2dd3-45c1-4fbe-b853-3c97b6508100","Type":"ContainerStarted","Data":"81b1de684ed4d30eb3c32f566b209bd56feaa47826bf079ca17b278bb5e435d0"}
Jul 01 17:17:52.416883 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:52.416491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kgg9j" event={"ID":"6bce2dd3-45c1-4fbe-b853-3c97b6508100","Type":"ContainerStarted","Data":"eb48d972575400e0957b163cd655e7095171230fd1cb59659905319c0daf3d7e"}
Jul 01 17:17:52.416883 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:52.416573    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:17:52.438025 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:52.437979    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kgg9j" podStartSLOduration=66.207083187 podStartE2EDuration="1m8.437964938s" totalImagesPullingTime="2.230881751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:17:52.436383377 +0000 UTC m=+102.842044164" watchObservedRunningTime="2026-07-01 17:17:52.437964938 +0000 UTC m=+102.843625714"
Jul 01 17:17:53.173439 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:53.173399    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:53.173661 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:53.173455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:53.173939 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:53.173920    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:53.175757 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:53.175739    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0cf27d80-b131-4905-8ec0-6a15ea46fc8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhkql\" (UID: \"0cf27d80-b131-4905-8ec0-6a15ea46fc8e\") " pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:53.274659 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:53.274621    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhkql"
Jul 01 17:17:53.423797 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:53.423729    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhkql"]
Jul 01 17:17:53.426485 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:17:53.426459    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf27d80_b131_4905_8ec0_6a15ea46fc8e.slice/crio-9b99c8dd0c531b551cfafd853f94eef41b468eec34bc28f8a5b5e2912f378d82 WatchSource:0}: Error finding container 9b99c8dd0c531b551cfafd853f94eef41b468eec34bc28f8a5b5e2912f378d82: Status 404 returned error can't find the container with id 9b99c8dd0c531b551cfafd853f94eef41b468eec34bc28f8a5b5e2912f378d82
Jul 01 17:17:54.426549 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:54.426453    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhkql" event={"ID":"0cf27d80-b131-4905-8ec0-6a15ea46fc8e","Type":"ContainerStarted","Data":"9b99c8dd0c531b551cfafd853f94eef41b468eec34bc28f8a5b5e2912f378d82"}
Jul 01 17:17:58.442749 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:58.442666    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhkql" event={"ID":"0cf27d80-b131-4905-8ec0-6a15ea46fc8e","Type":"ContainerStarted","Data":"32907a54c6886c778686c58720a56b24364e5f26d33064394d89d81d8fb97a36"}
Jul 01 17:17:58.442749 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:17:58.442723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhkql" event={"ID":"0cf27d80-b131-4905-8ec0-6a15ea46fc8e","Type":"ContainerStarted","Data":"e19111b380864f18abbbfbcd08a9830b3a1bd9ff05d24bc9edc3bb649f8e1e5c"}
Jul 01 17:18:00.257068 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.257032    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5499b6fb8f-skzxj"]
Jul 01 17:18:00.267165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.267133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-skzxj"
Jul 01 17:18:00.270817 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.269398    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-skzxj"]
Jul 01 17:18:00.271844 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.271446    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 01 17:18:00.271844 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.271558    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 01 17:18:00.271844 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.271702    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gkhjt\""
Jul 01 17:18:00.330754 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.330718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5xh4\" (UniqueName: \"kubernetes.io/projected/38413f5e-8ad5-4def-9fed-a74c356789d7-kube-api-access-p5xh4\") pod \"downloads-5499b6fb8f-skzxj\" (UID: \"38413f5e-8ad5-4def-9fed-a74c356789d7\") " pod="openshift-console/downloads-5499b6fb8f-skzxj"
Jul 01 17:18:00.431386 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.431357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xh4\" (UniqueName: \"kubernetes.io/projected/38413f5e-8ad5-4def-9fed-a74c356789d7-kube-api-access-p5xh4\") pod \"downloads-5499b6fb8f-skzxj\" (UID: \"38413f5e-8ad5-4def-9fed-a74c356789d7\") " pod="openshift-console/downloads-5499b6fb8f-skzxj"
Jul 01 17:18:00.440562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.440537    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xh4\" (UniqueName: \"kubernetes.io/projected/38413f5e-8ad5-4def-9fed-a74c356789d7-kube-api-access-p5xh4\") pod \"downloads-5499b6fb8f-skzxj\" (UID: \"38413f5e-8ad5-4def-9fed-a74c356789d7\") " pod="openshift-console/downloads-5499b6fb8f-skzxj"
Jul 01 17:18:00.580033 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.579913    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5499b6fb8f-skzxj"
Jul 01 17:18:00.710697 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:00.710659    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5499b6fb8f-skzxj"]
Jul 01 17:18:00.713970 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:18:00.713932    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38413f5e_8ad5_4def_9fed_a74c356789d7.slice/crio-5793fbdff8ebbc034ecad111a9ac2cd472b6a78387193a9da26d1f370ef42150 WatchSource:0}: Error finding container 5793fbdff8ebbc034ecad111a9ac2cd472b6a78387193a9da26d1f370ef42150: Status 404 returned error can't find the container with id 5793fbdff8ebbc034ecad111a9ac2cd472b6a78387193a9da26d1f370ef42150
Jul 01 17:18:01.455025 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:01.454985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhkql" event={"ID":"0cf27d80-b131-4905-8ec0-6a15ea46fc8e","Type":"ContainerStarted","Data":"15262e1e0ad64efad96d77b366239895ca47f059564e3cc341c396ef83b608f7"}
Jul 01 17:18:01.456011 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:01.455987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-skzxj" event={"ID":"38413f5e-8ad5-4def-9fed-a74c356789d7","Type":"ContainerStarted","Data":"5793fbdff8ebbc034ecad111a9ac2cd472b6a78387193a9da26d1f370ef42150"}
Jul 01 17:18:01.476846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:01.476798    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vhkql" podStartSLOduration=17.596574 podStartE2EDuration="24.47678395s" totalImagesPullingTime="6.88020995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:17:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:18:01.475222387 +0000 UTC m=+111.880883164" watchObservedRunningTime="2026-07-01 17:18:01.47678395 +0000 UTC m=+111.882444727"
Jul 01 17:18:02.422661 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:02.422624    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kgg9j"
Jul 01 17:18:09.194256 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.194203    2576 patch_prober.go:28] interesting pod/image-registry-768d4f548c-ftpw7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 01 17:18:09.194714 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.194291    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7" podUID="f793d210-92d9-4705-882c-064c9fdf2c23" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:18:09.280527 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.280496    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"]
Jul 01 17:18:09.282937 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.282915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.287236 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.287191    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 01 17:18:09.287236 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.287216    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 01 17:18:09.287436 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.287216    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 01 17:18:09.287436 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.287284    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 01 17:18:09.287436 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.287264    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 01 17:18:09.287436 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.287315    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l7z2w\""
Jul 01 17:18:09.298351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.297106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"]
Jul 01 17:18:09.400090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.400055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.400270 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.400100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pszwj\" (UniqueName: \"kubernetes.io/projected/6edd6a76-0086-4102-872a-62e611e07a98-kube-api-access-pszwj\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.400270 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.400181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.400270 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.400242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6edd6a76-0086-4102-872a-62e611e07a98-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.501198 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.501155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6edd6a76-0086-4102-872a-62e611e07a98-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.501382 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.501228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.501382 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.501257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pszwj\" (UniqueName: \"kubernetes.io/projected/6edd6a76-0086-4102-872a-62e611e07a98-kube-api-access-pszwj\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.501382 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.501311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.501589 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:18:09.501449    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 01 17:18:09.501589 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:18:09.501540    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls podName:6edd6a76-0086-4102-872a-62e611e07a98 nodeName:}" failed. No retries permitted until 2026-07-01 17:18:10.001505718 +0000 UTC m=+120.407166474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-nvcrd" (UID: "6edd6a76-0086-4102-872a-62e611e07a98") : secret "prometheus-operator-tls" not found
Jul 01 17:18:09.502025 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.501988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6edd6a76-0086-4102-872a-62e611e07a98-metrics-client-ca\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.504097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.504070    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:09.509632 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:09.509611    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pszwj\" (UniqueName: \"kubernetes.io/projected/6edd6a76-0086-4102-872a-62e611e07a98-kube-api-access-pszwj\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:10.005611 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:10.005564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:10.005765 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:18:10.005698    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 01 17:18:10.005823 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:18:10.005768    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls podName:6edd6a76-0086-4102-872a-62e611e07a98 nodeName:}" failed. No retries permitted until 2026-07-01 17:18:11.005747068 +0000 UTC m=+121.411407822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls") pod "prometheus-operator-7b7d896497-nvcrd" (UID: "6edd6a76-0086-4102-872a-62e611e07a98") : secret "prometheus-operator-tls" not found
Jul 01 17:18:10.411952 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:10.411867    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:18:11.013987 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:11.013947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:11.016617 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:11.016584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6edd6a76-0086-4102-872a-62e611e07a98-prometheus-operator-tls\") pod \"prometheus-operator-7b7d896497-nvcrd\" (UID: \"6edd6a76-0086-4102-872a-62e611e07a98\") " pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:11.096480 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:11.096442    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l7z2w\""
Jul 01 17:18:11.104816 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:11.104788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"
Jul 01 17:18:14.312601 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:14.312559    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" podUID="c759749d-5561-4f8b-9874-1c65130746ad" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 17:18:17.794836 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:17.794809    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7b7d896497-nvcrd"]
Jul 01 17:18:17.797673 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:18:17.797644    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edd6a76_0086_4102_872a_62e611e07a98.slice/crio-d4e388dfc6ee094ffd19404e8843ecd9130603a9c2f0509f8855fda0415caab2 WatchSource:0}: Error finding container d4e388dfc6ee094ffd19404e8843ecd9130603a9c2f0509f8855fda0415caab2: Status 404 returned error can't find the container with id d4e388dfc6ee094ffd19404e8843ecd9130603a9c2f0509f8855fda0415caab2
Jul 01 17:18:18.517029 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:18.516978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5499b6fb8f-skzxj" event={"ID":"38413f5e-8ad5-4def-9fed-a74c356789d7","Type":"ContainerStarted","Data":"86cff3d1b5308bd212dc000c6b0647fe76fdedc38b8dc290713c4dec3ee764f0"}
Jul 01 17:18:18.517224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:18.517171    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5499b6fb8f-skzxj"
Jul 01 17:18:18.518425 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:18.518394    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd" event={"ID":"6edd6a76-0086-4102-872a-62e611e07a98","Type":"ContainerStarted","Data":"d4e388dfc6ee094ffd19404e8843ecd9130603a9c2f0509f8855fda0415caab2"}
Jul 01 17:18:18.546056 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:18.546025    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5499b6fb8f-skzxj"
Jul 01 17:18:18.586044 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:18.585986    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5499b6fb8f-skzxj" podStartSLOduration=1.569586826 podStartE2EDuration="18.585967897s" totalImagesPullingTime="17.016381071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:18:18.543981287 +0000 UTC m=+128.949642064" watchObservedRunningTime="2026-07-01 17:18:18.585967897 +0000 UTC m=+128.991628711"
Jul 01 17:18:19.991204 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:19.991159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:18:19.994437 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:19.994377    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7ba0dbe-db80-4c86-afc4-545af38d30d0-metrics-certs\") pod \"network-metrics-daemon-tb5ht\" (UID: \"c7ba0dbe-db80-4c86-afc4-545af38d30d0\") " pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:18:20.019819 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:20.019677    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-768d4f548c-ftpw7"]
Jul 01 17:18:20.161128 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:20.161087    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gs6qb\""
Jul 01 17:18:20.168953 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:20.168908    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tb5ht"
Jul 01 17:18:20.391636 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:20.391600    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tb5ht"]
Jul 01 17:18:20.528814 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:18:20.528745    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ba0dbe_db80_4c86_afc4_545af38d30d0.slice/crio-bfef6d0dd26081a394935e0f7df9dcf238c36bbdef0a349b14f57602a3f82478 WatchSource:0}: Error finding container bfef6d0dd26081a394935e0f7df9dcf238c36bbdef0a349b14f57602a3f82478: Status 404 returned error can't find the container with id bfef6d0dd26081a394935e0f7df9dcf238c36bbdef0a349b14f57602a3f82478
Jul 01 17:18:21.533674 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:21.533626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd" event={"ID":"6edd6a76-0086-4102-872a-62e611e07a98","Type":"ContainerStarted","Data":"ecddf4e72e738cc1c63d974b0c64194858439c0116316a5eee2799d07e91c505"}
Jul 01 17:18:21.534251 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:21.533689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd" event={"ID":"6edd6a76-0086-4102-872a-62e611e07a98","Type":"ContainerStarted","Data":"78feace45780cc69b026e6fe87f897787f0d7fe3021791c00cba1aa74861415e"}
Jul 01 17:18:21.534989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:21.534954    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb5ht" event={"ID":"c7ba0dbe-db80-4c86-afc4-545af38d30d0","Type":"ContainerStarted","Data":"bfef6d0dd26081a394935e0f7df9dcf238c36bbdef0a349b14f57602a3f82478"}
Jul 01 17:18:21.554000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:21.553944    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7b7d896497-nvcrd" podStartSLOduration=9.806331044 podStartE2EDuration="12.55392637s" totalImagesPullingTime="2.747595326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:18:21.552274478 +0000 UTC m=+131.957935254" watchObservedRunningTime="2026-07-01 17:18:21.55392637 +0000 UTC m=+131.959587146"
Jul 01 17:18:23.544488 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.544443    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb5ht" event={"ID":"c7ba0dbe-db80-4c86-afc4-545af38d30d0","Type":"ContainerStarted","Data":"f53483098304521a13424cfa0c99d5a5fe58d63d2ab41588eba2f7f49f15d1eb"}
Jul 01 17:18:23.544967 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.544558    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tb5ht" event={"ID":"c7ba0dbe-db80-4c86-afc4-545af38d30d0","Type":"ContainerStarted","Data":"0d0192a2b2bffa5ec385900c073e9a3cb65a3e94da898f0c6f8071aa4db91513"}
Jul 01 17:18:23.568165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.568107    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tb5ht" podStartSLOduration=131.454972749 podStartE2EDuration="2m13.568089796s" totalImagesPullingTime="2.113117047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:18:23.566488002 +0000 UTC m=+133.972148793" watchObservedRunningTime="2026-07-01 17:18:23.568089796 +0000 UTC m=+133.973750572"
Jul 01 17:18:23.759484 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.759434    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xbz58"]
Jul 01 17:18:23.785686 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.785651    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-gzrvs"]
Jul 01 17:18:23.785838 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.785792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.789215 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.789190    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 01 17:18:23.789362 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.789222    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w728g\""
Jul 01 17:18:23.789362 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.789192    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 01 17:18:23.789729 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.789705    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 01 17:18:23.809689 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.809631    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-gzrvs"]
Jul 01 17:18:23.809793 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.809756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:23.812655 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.812635    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 01 17:18:23.812811 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.812649    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 01 17:18:23.812901 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.812762    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 01 17:18:23.813027 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.813003    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7xmbc\""
Jul 01 17:18:23.920994 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.920954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkdgw\" (UniqueName: \"kubernetes.io/projected/64a2befe-28f1-4bbb-9d47-1ff5d1088269-kube-api-access-xkdgw\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921163 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921163 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:23.921163 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c856c3-297e-45fb-acfc-676730877495-metrics-client-ca\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:23.921308 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921170    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:23.921308 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-textfile\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921308 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-accelerators-collector-config\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921308 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:23.921308 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-tls\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-root\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921399    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-sys\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b1c856c3-297e-45fb-acfc-676730877495-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:23.921562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-wtmp\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921562 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a2befe-28f1-4bbb-9d47-1ff5d1088269-metrics-client-ca\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:23.921774 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:23.921613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl6f5\" (UniqueName: \"kubernetes.io/projected/b1c856c3-297e-45fb-acfc-676730877495-kube-api-access-bl6f5\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.022913 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.022876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.022932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.023097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.022965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c856c3-297e-45fb-acfc-676730877495-metrics-client-ca\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.023097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.022991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.023097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-textfile\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-accelerators-collector-config\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-tls\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-root\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:18:24.023133    2576 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-sys\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b1c856c3-297e-45fb-acfc-676730877495-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-wtmp\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a2befe-28f1-4bbb-9d47-1ff5d1088269-metrics-client-ca\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl6f5\" (UniqueName: \"kubernetes.io/projected/b1c856c3-297e-45fb-acfc-676730877495-kube-api-access-bl6f5\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.023396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkdgw\" (UniqueName: \"kubernetes.io/projected/64a2befe-28f1-4bbb-9d47-1ff5d1088269-kube-api-access-xkdgw\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023873 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023729    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-accelerators-collector-config\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023873 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023741    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-sys\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023873 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023805    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-textfile\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.023873 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1c856c3-297e-45fb-acfc-676730877495-metrics-client-ca\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.024069 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:18:24.023864    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-tls podName:b1c856c3-297e-45fb-acfc-676730877495 nodeName:}" failed. No retries permitted until 2026-07-01 17:18:24.523839644 +0000 UTC m=+134.929500401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-tls") pod "kube-state-metrics-845b954455-gzrvs" (UID: "b1c856c3-297e-45fb-acfc-676730877495") : secret "kube-state-metrics-tls" not found
Jul 01 17:18:24.024069 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.023977    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-wtmp\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.024069 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.024051    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b1c856c3-297e-45fb-acfc-676730877495-volume-directive-shadow\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.024278 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.024259    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/64a2befe-28f1-4bbb-9d47-1ff5d1088269-root\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.024382 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.024356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.024438 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.024365    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64a2befe-28f1-4bbb-9d47-1ff5d1088269-metrics-client-ca\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.025887 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.025848    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.026227 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.026202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.026299 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.026278    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/64a2befe-28f1-4bbb-9d47-1ff5d1088269-node-exporter-tls\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.035716 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.035688    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl6f5\" (UniqueName: \"kubernetes.io/projected/b1c856c3-297e-45fb-acfc-676730877495-kube-api-access-bl6f5\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.035825 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.035793    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkdgw\" (UniqueName: \"kubernetes.io/projected/64a2befe-28f1-4bbb-9d47-1ff5d1088269-kube-api-access-xkdgw\") pod \"node-exporter-xbz58\" (UID: \"64a2befe-28f1-4bbb-9d47-1ff5d1088269\") " pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.098068 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.097989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xbz58"
Jul 01 17:18:24.156616 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:18:24.156572    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a2befe_28f1_4bbb_9d47_1ff5d1088269.slice/crio-f2f0b5ce0e14dd7cb9059c5c709d6135a3d66158933d001c0722fea85db7354c WatchSource:0}: Error finding container f2f0b5ce0e14dd7cb9059c5c709d6135a3d66158933d001c0722fea85db7354c: Status 404 returned error can't find the container with id f2f0b5ce0e14dd7cb9059c5c709d6135a3d66158933d001c0722fea85db7354c
Jul 01 17:18:24.312142 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.312098    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" podUID="c759749d-5561-4f8b-9874-1c65130746ad" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 17:18:24.528608 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.528565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.531286 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.531250    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1c856c3-297e-45fb-acfc-676730877495-kube-state-metrics-tls\") pod \"kube-state-metrics-845b954455-gzrvs\" (UID: \"b1c856c3-297e-45fb-acfc-676730877495\") " pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.551116 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.551076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbz58" event={"ID":"64a2befe-28f1-4bbb-9d47-1ff5d1088269","Type":"ContainerStarted","Data":"f2f0b5ce0e14dd7cb9059c5c709d6135a3d66158933d001c0722fea85db7354c"}
Jul 01 17:18:24.720381 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.720339    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs"
Jul 01 17:18:24.900017 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:24.899984    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-845b954455-gzrvs"]
Jul 01 17:18:24.902734 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:18:24.902695    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c856c3_297e_45fb_acfc_676730877495.slice/crio-164fb4cf4c0a4c6437b41c08831ca17f0806918e83956099245494e7c608d244 WatchSource:0}: Error finding container 164fb4cf4c0a4c6437b41c08831ca17f0806918e83956099245494e7c608d244: Status 404 returned error can't find the container with id 164fb4cf4c0a4c6437b41c08831ca17f0806918e83956099245494e7c608d244
Jul 01 17:18:25.557574 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:25.557514    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs" event={"ID":"b1c856c3-297e-45fb-acfc-676730877495","Type":"ContainerStarted","Data":"164fb4cf4c0a4c6437b41c08831ca17f0806918e83956099245494e7c608d244"}
Jul 01 17:18:27.567138 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:27.567061    2576 generic.go:356] "Generic (PLEG): container finished" podID="64a2befe-28f1-4bbb-9d47-1ff5d1088269" containerID="c0ff1d2b28e80b4e187453620f1e72ac34879fb49b94d2c64e2c7f9c0e87e4c3" exitCode=0
Jul 01 17:18:27.567586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:27.567147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbz58" event={"ID":"64a2befe-28f1-4bbb-9d47-1ff5d1088269","Type":"ContainerDied","Data":"c0ff1d2b28e80b4e187453620f1e72ac34879fb49b94d2c64e2c7f9c0e87e4c3"}
Jul 01 17:18:27.569427 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:27.569401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs" event={"ID":"b1c856c3-297e-45fb-acfc-676730877495","Type":"ContainerStarted","Data":"612e6770d83d48ce55cfce4485a4b6557c698b99731b3ea8b7b312055c45c3a0"}
Jul 01 17:18:27.569551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:27.569464    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs" event={"ID":"b1c856c3-297e-45fb-acfc-676730877495","Type":"ContainerStarted","Data":"96fcf92a1bc1cac4a977c403e00d091d5f74a2267b916214c07a185081ca767d"}
Jul 01 17:18:27.569551 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:27.569493    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs" event={"ID":"b1c856c3-297e-45fb-acfc-676730877495","Type":"ContainerStarted","Data":"8fce4cc20c4c758d885ff845e17d88707e97c24cfc8ea5bc835d3ce6ad435402"}
Jul 01 17:18:27.606965 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:27.606903    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-845b954455-gzrvs" podStartSLOduration=2.5013023 podStartE2EDuration="4.606883067s" totalImagesPullingTime="2.105580767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:18:27.605647882 +0000 UTC m=+138.011308659" watchObservedRunningTime="2026-07-01 17:18:27.606883067 +0000 UTC m=+138.012543843"
Jul 01 17:18:28.575809 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:28.575767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbz58" event={"ID":"64a2befe-28f1-4bbb-9d47-1ff5d1088269","Type":"ContainerStarted","Data":"2352cea78e2ccbcaaed8e6ce52a947236d34ef07ced3fed76a75b88971652d70"}
Jul 01 17:18:28.576239 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:28.575831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xbz58" event={"ID":"64a2befe-28f1-4bbb-9d47-1ff5d1088269","Type":"ContainerStarted","Data":"5fd52ea5e03aa70cebedc6740569d4eb0174678a830e7a6105f17f584b420319"}
Jul 01 17:18:28.600803 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:28.600755    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xbz58" podStartSLOduration=2.635947051 podStartE2EDuration="5.600739153s" totalImagesPullingTime="2.849600503s" totalInitContainerRuntime="115.191599ms" isStatefulPod=true podCreationTimestamp="2026-07-01 17:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:18:28.599491389 +0000 UTC m=+139.005152206" watchObservedRunningTime="2026-07-01 17:18:28.600739153 +0000 UTC m=+139.006399963"
Jul 01 17:18:34.311923 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:34.311882    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" podUID="c759749d-5561-4f8b-9874-1c65130746ad" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 01 17:18:34.312549 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:34.311961    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk"
Jul 01 17:18:34.312663 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:34.312623    2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"ec11c0b750a500739a39cafadb30671ee85e3d949986b3ecd837860b9ea9e08e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 01 17:18:34.312723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:34.312693    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" podUID="c759749d-5561-4f8b-9874-1c65130746ad" containerName="service-proxy" containerID="cri-o://ec11c0b750a500739a39cafadb30671ee85e3d949986b3ecd837860b9ea9e08e" gracePeriod=30
Jul 01 17:18:34.599750 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:34.599607    2576 generic.go:356] "Generic (PLEG): container finished" podID="c759749d-5561-4f8b-9874-1c65130746ad" containerID="ec11c0b750a500739a39cafadb30671ee85e3d949986b3ecd837860b9ea9e08e" exitCode=2
Jul 01 17:18:34.599750 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:34.599643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" event={"ID":"c759749d-5561-4f8b-9874-1c65130746ad","Type":"ContainerDied","Data":"ec11c0b750a500739a39cafadb30671ee85e3d949986b3ecd837860b9ea9e08e"}
Jul 01 17:18:34.599750 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:34.599707    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fb7994cf9-bqbqk" event={"ID":"c759749d-5561-4f8b-9874-1c65130746ad","Type":"ContainerStarted","Data":"0d0c3cb72b7d2538d3ffe5e559d10a1c1a80b7e035e6ecac9394f2084cde6d4b"}
Jul 01 17:18:45.049601 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.049556    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7" podUID="f793d210-92d9-4705-882c-064c9fdf2c23" containerName="registry" containerID="cri-o://d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666" gracePeriod=30
Jul 01 17:18:45.286572 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.286549    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:18:45.391242 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391155    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-installation-pull-secrets\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391242 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391214    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-registry-certificates\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391247    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-trusted-ca\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391267    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-image-registry-private-configuration\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391288    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjh22\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-kube-api-access-gjh22\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391310    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-bound-sa-token\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391344    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f793d210-92d9-4705-882c-064c9fdf2c23-ca-trust-extracted\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391370    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") pod \"f793d210-92d9-4705-882c-064c9fdf2c23\" (UID: \"f793d210-92d9-4705-882c-064c9fdf2c23\") "
Jul 01 17:18:45.391802 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391737    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:18:45.391802 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.391754    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:18:45.393839 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.393782    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:18:45.394038 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.394002    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-kube-api-access-gjh22" (OuterVolumeSpecName: "kube-api-access-gjh22") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "kube-api-access-gjh22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:18:45.394151 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.394096    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:18:45.394151 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.394102    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:18:45.394227 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.394140    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:18:45.399988 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.399960    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f793d210-92d9-4705-882c-064c9fdf2c23-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f793d210-92d9-4705-882c-064c9fdf2c23" (UID: "f793d210-92d9-4705-882c-064c9fdf2c23"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:18:45.492204 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492171    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-installation-pull-secrets\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.492204 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492200    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-registry-certificates\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.492204 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492210    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f793d210-92d9-4705-882c-064c9fdf2c23-trusted-ca\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.492407 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492221    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f793d210-92d9-4705-882c-064c9fdf2c23-image-registry-private-configuration\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.492407 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492232    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjh22\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-kube-api-access-gjh22\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.492407 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492240    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-bound-sa-token\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.492407 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492249    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f793d210-92d9-4705-882c-064c9fdf2c23-ca-trust-extracted\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.492407 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.492257    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f793d210-92d9-4705-882c-064c9fdf2c23-registry-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:18:45.639075 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.639041    2576 generic.go:356] "Generic (PLEG): container finished" podID="f793d210-92d9-4705-882c-064c9fdf2c23" containerID="d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666" exitCode=0
Jul 01 17:18:45.639232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.639088    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7"
Jul 01 17:18:45.639232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.639115    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7" event={"ID":"f793d210-92d9-4705-882c-064c9fdf2c23","Type":"ContainerDied","Data":"d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666"}
Jul 01 17:18:45.639232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.639170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-768d4f548c-ftpw7" event={"ID":"f793d210-92d9-4705-882c-064c9fdf2c23","Type":"ContainerDied","Data":"939e139424229bc9dc16ededcf28e765d14b2b68c1f9b64f5b361bb86f9adea7"}
Jul 01 17:18:45.639232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.639192    2576 scope.go:117] "RemoveContainer" containerID="d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666"
Jul 01 17:18:45.647369 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.647332    2576 scope.go:117] "RemoveContainer" containerID="d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666"
Jul 01 17:18:45.647697 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:18:45.647678    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666\": container with ID starting with d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666 not found: ID does not exist" containerID="d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666"
Jul 01 17:18:45.647751 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.647706    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666"} err="failed to get container status \"d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666\": rpc error: code = NotFound desc = could not find container \"d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666\": container with ID starting with d4dd6310c7cab1f4fb1c2141d681eacda39ac00c22c7e865319868386d251666 not found: ID does not exist"
Jul 01 17:18:45.660673 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.660651    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-768d4f548c-ftpw7"]
Jul 01 17:18:45.664916 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:45.664897    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-768d4f548c-ftpw7"]
Jul 01 17:18:46.048815 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:18:46.048786    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f793d210-92d9-4705-882c-064c9fdf2c23" path="/var/lib/kubelet/pods/f793d210-92d9-4705-882c-064c9fdf2c23/volumes"
Jul 01 17:21:09.970095 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:09.970068    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:21:09.972295 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:09.972276    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 01 17:21:09.972532 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:09.972503    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:21:52.609275 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.609242    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"]
Jul 01 17:21:52.609740 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.609496    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f793d210-92d9-4705-882c-064c9fdf2c23" containerName="registry"
Jul 01 17:21:52.609740 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.609506    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f793d210-92d9-4705-882c-064c9fdf2c23" containerName="registry"
Jul 01 17:21:52.609740 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.609573    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f793d210-92d9-4705-882c-064c9fdf2c23" containerName="registry"
Jul 01 17:21:52.614701 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.614675    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.618160 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.618138    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z4zjr\""
Jul 01 17:21:52.618288 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.618139    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 01 17:21:52.619300 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.619283    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 01 17:21:52.630784 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.630760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"]
Jul 01 17:21:52.688151 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.688119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.688326 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.688189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nzh8\" (UniqueName: \"kubernetes.io/projected/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-kube-api-access-5nzh8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.688326 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.688224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.789358 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.789317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nzh8\" (UniqueName: \"kubernetes.io/projected/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-kube-api-access-5nzh8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.789563 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.789369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.789563 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.789396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.789797 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.789781    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.789832 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.789815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.799275 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.799253    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nzh8\" (UniqueName: \"kubernetes.io/projected/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-kube-api-access-5nzh8\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:52.923041 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:52.922957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:21:53.052201 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:53.052172    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"]
Jul 01 17:21:53.055486 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:21:53.055461    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecaf0d1_bce1_4ac1_9e03_7e45611bffca.slice/crio-356471d94f31094e930ae8585687c970a88688fbb1465ec2968f8507ee6507c8 WatchSource:0}: Error finding container 356471d94f31094e930ae8585687c970a88688fbb1465ec2968f8507ee6507c8: Status 404 returned error can't find the container with id 356471d94f31094e930ae8585687c970a88688fbb1465ec2968f8507ee6507c8
Jul 01 17:21:53.057443 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:53.057425    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:21:53.258641 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:21:53.258608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr" event={"ID":"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca","Type":"ContainerStarted","Data":"356471d94f31094e930ae8585687c970a88688fbb1465ec2968f8507ee6507c8"}
Jul 01 17:22:01.288837 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:01.288803    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerID="690bbadc1338fb277b017cec562515e10403953ba102306ae96d2a245a86fca1" exitCode=0
Jul 01 17:22:01.289223 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:01.288893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr" event={"ID":"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca","Type":"ContainerDied","Data":"690bbadc1338fb277b017cec562515e10403953ba102306ae96d2a245a86fca1"}
Jul 01 17:22:04.302762 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:04.302732    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerID="bda1f5d5b0263d511933d747004cf6db29f6b1032321bfe9a1904d9c85fde0e7" exitCode=0
Jul 01 17:22:04.303122 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:04.302771    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr" event={"ID":"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca","Type":"ContainerDied","Data":"bda1f5d5b0263d511933d747004cf6db29f6b1032321bfe9a1904d9c85fde0e7"}
Jul 01 17:22:19.356263 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:19.356224    2576 generic.go:356] "Generic (PLEG): container finished" podID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerID="d9bd7fb814d71ebace00d5210efc3d318c32039de7e662d76ea6eb9a5f142d46" exitCode=0
Jul 01 17:22:19.356667 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:19.356282    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr" event={"ID":"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca","Type":"ContainerDied","Data":"d9bd7fb814d71ebace00d5210efc3d318c32039de7e662d76ea6eb9a5f142d46"}
Jul 01 17:22:20.480062 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.480032    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:22:20.612234 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.612140    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-bundle\") pod \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") "
Jul 01 17:22:20.612234 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.612196    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nzh8\" (UniqueName: \"kubernetes.io/projected/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-kube-api-access-5nzh8\") pod \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") "
Jul 01 17:22:20.612234 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.612223    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-util\") pod \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\" (UID: \"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca\") "
Jul 01 17:22:20.613018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.612987    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-bundle" (OuterVolumeSpecName: "bundle") pod "0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" (UID: "0ecaf0d1-bce1-4ac1-9e03-7e45611bffca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:22:20.614361 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.614341    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-kube-api-access-5nzh8" (OuterVolumeSpecName: "kube-api-access-5nzh8") pod "0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" (UID: "0ecaf0d1-bce1-4ac1-9e03-7e45611bffca"). InnerVolumeSpecName "kube-api-access-5nzh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:22:20.618153 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.618128    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-util" (OuterVolumeSpecName: "util") pod "0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" (UID: "0ecaf0d1-bce1-4ac1-9e03-7e45611bffca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:22:20.713054 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.713017    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-bundle\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:22:20.713054 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.713049    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nzh8\" (UniqueName: \"kubernetes.io/projected/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-kube-api-access-5nzh8\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:22:20.713054 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:20.713063    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ecaf0d1-bce1-4ac1-9e03-7e45611bffca-util\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:22:21.364010 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:21.363982    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr"
Jul 01 17:22:21.364168 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:21.363984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589rzvzr" event={"ID":"0ecaf0d1-bce1-4ac1-9e03-7e45611bffca","Type":"ContainerDied","Data":"356471d94f31094e930ae8585687c970a88688fbb1465ec2968f8507ee6507c8"}
Jul 01 17:22:21.364168 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:21.364101    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="356471d94f31094e930ae8585687c970a88688fbb1465ec2968f8507ee6507c8"
Jul 01 17:22:26.228148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228114    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"]
Jul 01 17:22:26.228511 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228342    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerName="util"
Jul 01 17:22:26.228511 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228353    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerName="util"
Jul 01 17:22:26.228511 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228363    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerName="pull"
Jul 01 17:22:26.228511 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228368    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerName="pull"
Jul 01 17:22:26.228511 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228388    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerName="extract"
Jul 01 17:22:26.228511 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228395    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerName="extract"
Jul 01 17:22:26.228511 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.228439    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ecaf0d1-bce1-4ac1-9e03-7e45611bffca" containerName="extract"
Jul 01 17:22:26.230423 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.230405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.233572 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.233549    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 01 17:22:26.233900 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.233844    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 01 17:22:26.234077 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.234056    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 01 17:22:26.234140 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.234056    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-b6csz\""
Jul 01 17:22:26.246196 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.246174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"]
Jul 01 17:22:26.353732 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.353694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrlrf\" (UniqueName: \"kubernetes.io/projected/70ba4934-f0ca-49f5-b4e5-0862885aece2-kube-api-access-zrlrf\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh\" (UID: \"70ba4934-f0ca-49f5-b4e5-0862885aece2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.353732 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.353736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/70ba4934-f0ca-49f5-b4e5-0862885aece2-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh\" (UID: \"70ba4934-f0ca-49f5-b4e5-0862885aece2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.455054 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.455010    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrlrf\" (UniqueName: \"kubernetes.io/projected/70ba4934-f0ca-49f5-b4e5-0862885aece2-kube-api-access-zrlrf\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh\" (UID: \"70ba4934-f0ca-49f5-b4e5-0862885aece2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.455054 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.455054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/70ba4934-f0ca-49f5-b4e5-0862885aece2-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh\" (UID: \"70ba4934-f0ca-49f5-b4e5-0862885aece2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.457377 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.457358    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/70ba4934-f0ca-49f5-b4e5-0862885aece2-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh\" (UID: \"70ba4934-f0ca-49f5-b4e5-0862885aece2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.464289 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.464267    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrlrf\" (UniqueName: \"kubernetes.io/projected/70ba4934-f0ca-49f5-b4e5-0862885aece2-kube-api-access-zrlrf\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh\" (UID: \"70ba4934-f0ca-49f5-b4e5-0862885aece2\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.539221 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.539124    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:26.679790 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:26.679757    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"]
Jul 01 17:22:26.683071 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:22:26.683046    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ba4934_f0ca_49f5_b4e5_0862885aece2.slice/crio-d1bfb27337b8a322d1ab4ce20a2a118b8186af71dffa3f107442f790a6112743 WatchSource:0}: Error finding container d1bfb27337b8a322d1ab4ce20a2a118b8186af71dffa3f107442f790a6112743: Status 404 returned error can't find the container with id d1bfb27337b8a322d1ab4ce20a2a118b8186af71dffa3f107442f790a6112743
Jul 01 17:22:27.387451 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:27.387417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh" event={"ID":"70ba4934-f0ca-49f5-b4e5-0862885aece2","Type":"ContainerStarted","Data":"d1bfb27337b8a322d1ab4ce20a2a118b8186af71dffa3f107442f790a6112743"}
Jul 01 17:22:30.110709 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.110680    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-lqkzs"]
Jul 01 17:22:30.113255 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.113238    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.116212 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.116184    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 01 17:22:30.116212 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.116203    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 01 17:22:30.116366 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.116206    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-f954g\""
Jul 01 17:22:30.125272 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.125239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-lqkzs"]
Jul 01 17:22:30.181568 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.181514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c7bb54c8-0a61-4d9c-b74a-b90036128970-cabundle0\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.181568 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.181578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqdn\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-kube-api-access-4zqdn\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.181788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.181611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.282765 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.282730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c7bb54c8-0a61-4d9c-b74a-b90036128970-cabundle0\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.282945 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.282775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqdn\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-kube-api-access-4zqdn\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.282945 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.282835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.283028 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.282953    2576 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 01 17:22:30.283028 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.282978    2576 secret.go:281] references non-existent secret key: ca.crt
Jul 01 17:22:30.283028 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.282989    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 01 17:22:30.283028 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.283005    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-lqkzs: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 01 17:22:30.283214 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.283075    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates podName:c7bb54c8-0a61-4d9c-b74a-b90036128970 nodeName:}" failed. No retries permitted until 2026-07-01 17:22:30.783054644 +0000 UTC m=+381.188715420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates") pod "keda-operator-68bff59c-lqkzs" (UID: "c7bb54c8-0a61-4d9c-b74a-b90036128970") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 01 17:22:30.283543 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.283501    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c7bb54c8-0a61-4d9c-b74a-b90036128970-cabundle0\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.291121 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.291096    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqdn\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-kube-api-access-4zqdn\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.316256 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.316223    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"]
Jul 01 17:22:30.318499 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.318479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.321344 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.321317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 01 17:22:30.330359 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.330333    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"]
Jul 01 17:22:30.383986 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.383901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/26060df7-2152-44db-baff-4d5431d091a6-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.383986 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.383946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-certificates\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.383986 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.383962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmr62\" (UniqueName: \"kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-kube-api-access-lmr62\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.405444 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.405403    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh" event={"ID":"70ba4934-f0ca-49f5-b4e5-0862885aece2","Type":"ContainerStarted","Data":"c5e2fde8759984a4fdcab1ffcebb5c02e3d22e0bb7072ce2b7087a1123ea5a21"}
Jul 01 17:22:30.405626 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.405509    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:30.426897 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.426845    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh" podStartSLOduration=1.689729712 podStartE2EDuration="4.4268292s" totalImagesPullingTime="2.737099488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:22:30.425501289 +0000 UTC m=+380.831162067" watchObservedRunningTime="2026-07-01 17:22:30.4268292 +0000 UTC m=+380.832489976"
Jul 01 17:22:30.484856 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.484824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmr62\" (UniqueName: \"kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-kube-api-access-lmr62\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.485026 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.484955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/26060df7-2152-44db-baff-4d5431d091a6-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.485026 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.485005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-certificates\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.485112 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.485092    2576 secret.go:281] references non-existent secret key: tls.crt
Jul 01 17:22:30.485112 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.485108    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 01 17:22:30.485170 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.485124    2576 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 01 17:22:30.485170 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.485141    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 01 17:22:30.485229 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.485201    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-certificates podName:26060df7-2152-44db-baff-4d5431d091a6 nodeName:}" failed. No retries permitted until 2026-07-01 17:22:30.985184239 +0000 UTC m=+381.390844999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-certificates") pod "keda-metrics-apiserver-559f75f947-lmgz8" (UID: "26060df7-2152-44db-baff-4d5431d091a6") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 01 17:22:30.485443 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.485424    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/26060df7-2152-44db-baff-4d5431d091a6-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.494365 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.494341    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmr62\" (UniqueName: \"kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-kube-api-access-lmr62\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.787894 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.787848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:30.788080 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.788015    2576 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 01 17:22:30.788080 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.788040    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-lqkzs: secret "keda-operator-certs" not found
Jul 01 17:22:30.788202 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:22:30.788104    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates podName:c7bb54c8-0a61-4d9c-b74a-b90036128970 nodeName:}" failed. No retries permitted until 2026-07-01 17:22:31.788084808 +0000 UTC m=+382.193745580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates") pod "keda-operator-68bff59c-lqkzs" (UID: "c7bb54c8-0a61-4d9c-b74a-b90036128970") : secret "keda-operator-certs" not found
Jul 01 17:22:30.990307 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.990271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-certificates\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:30.993128 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:30.993100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/26060df7-2152-44db-baff-4d5431d091a6-certificates\") pod \"keda-metrics-apiserver-559f75f947-lmgz8\" (UID: \"26060df7-2152-44db-baff-4d5431d091a6\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:31.228702 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:31.228660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:31.373080 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:31.373045    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"]
Jul 01 17:22:31.376389 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:22:31.376353    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26060df7_2152_44db_baff_4d5431d091a6.slice/crio-4ffd5e92a705296fce6a807c089a3af151a3f7f1970530bd1a5657eb2b4d17e9 WatchSource:0}: Error finding container 4ffd5e92a705296fce6a807c089a3af151a3f7f1970530bd1a5657eb2b4d17e9: Status 404 returned error can't find the container with id 4ffd5e92a705296fce6a807c089a3af151a3f7f1970530bd1a5657eb2b4d17e9
Jul 01 17:22:31.417350 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:31.417287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8" event={"ID":"26060df7-2152-44db-baff-4d5431d091a6","Type":"ContainerStarted","Data":"4ffd5e92a705296fce6a807c089a3af151a3f7f1970530bd1a5657eb2b4d17e9"}
Jul 01 17:22:31.796992 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:31.796953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:31.799348 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:31.799318    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bb54c8-0a61-4d9c-b74a-b90036128970-certificates\") pod \"keda-operator-68bff59c-lqkzs\" (UID: \"c7bb54c8-0a61-4d9c-b74a-b90036128970\") " pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:31.922817 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:31.922777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:32.048350 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:22:32.048275    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bb54c8_0a61_4d9c_b74a_b90036128970.slice/crio-aabb41d169c5e670f5daec9eaefa5810ae94c347372c3ba2731f6fe396007f39 WatchSource:0}: Error finding container aabb41d169c5e670f5daec9eaefa5810ae94c347372c3ba2731f6fe396007f39: Status 404 returned error can't find the container with id aabb41d169c5e670f5daec9eaefa5810ae94c347372c3ba2731f6fe396007f39
Jul 01 17:22:32.048463 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:32.048359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-lqkzs"]
Jul 01 17:22:32.423459 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:32.423374    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-lqkzs" event={"ID":"c7bb54c8-0a61-4d9c-b74a-b90036128970","Type":"ContainerStarted","Data":"aabb41d169c5e670f5daec9eaefa5810ae94c347372c3ba2731f6fe396007f39"}
Jul 01 17:22:35.436271 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:35.436237    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8" event={"ID":"26060df7-2152-44db-baff-4d5431d091a6","Type":"ContainerStarted","Data":"2220d41c990588014b517ab44205130cf0f7617cb066a95613331df8973a2e0d"}
Jul 01 17:22:35.436781 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:35.436353    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:35.454846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:35.454775    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8" podStartSLOduration=2.427979586 podStartE2EDuration="5.454758168s" totalImagesPullingTime="3.026778582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:22:35.453777648 +0000 UTC m=+385.859438460" watchObservedRunningTime="2026-07-01 17:22:35.454758168 +0000 UTC m=+385.860418945"
Jul 01 17:22:36.441122 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:36.441083    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-lqkzs" event={"ID":"c7bb54c8-0a61-4d9c-b74a-b90036128970","Type":"ContainerStarted","Data":"f044037d27a687466c86d5761d240b4ca492309477d2474b87017a0a0554b2ba"}
Jul 01 17:22:36.441493 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:36.441252    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:22:36.460825 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:36.460779    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-lqkzs" podStartSLOduration=3.027294833 podStartE2EDuration="6.46075821s" totalImagesPullingTime="3.433463377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:22:36.458958361 +0000 UTC m=+386.864619137" watchObservedRunningTime="2026-07-01 17:22:36.46075821 +0000 UTC m=+386.866419023"
Jul 01 17:22:46.445292 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:46.445257    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-lmgz8"
Jul 01 17:22:51.420230 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:51.420197    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-6cqjh"
Jul 01 17:22:57.447920 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:22:57.447885    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-lqkzs"
Jul 01 17:23:38.541213 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.541133    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-848c499849-vgmzr"]
Jul 01 17:23:38.543283 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.543268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.547385 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.547365    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 17:23:38.548332 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.548315    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 17:23:38.548708 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.548693    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 01 17:23:38.548792 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.548766    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qr799\""
Jul 01 17:23:38.560379 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.560351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-848c499849-vgmzr"]
Jul 01 17:23:38.609982 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.609952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2wz\" (UniqueName: \"kubernetes.io/projected/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-kube-api-access-wm2wz\") pod \"kserve-controller-manager-848c499849-vgmzr\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") " pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.610166 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.610048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-cert\") pod \"kserve-controller-manager-848c499849-vgmzr\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") " pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.710414 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.710382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2wz\" (UniqueName: \"kubernetes.io/projected/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-kube-api-access-wm2wz\") pod \"kserve-controller-manager-848c499849-vgmzr\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") " pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.710641 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.710466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-cert\") pod \"kserve-controller-manager-848c499849-vgmzr\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") " pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.712871 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.712845    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-cert\") pod \"kserve-controller-manager-848c499849-vgmzr\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") " pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.719605 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.719585    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2wz\" (UniqueName: \"kubernetes.io/projected/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-kube-api-access-wm2wz\") pod \"kserve-controller-manager-848c499849-vgmzr\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") " pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.853144 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.853033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:38.978303 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:38.978275    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-848c499849-vgmzr"]
Jul 01 17:23:38.980920 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:23:38.980896    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427e33b1_7ec9_4656_9e2c_a75ae42a44c3.slice/crio-89022463ae98cdd739b1e811a7cf996695c85ffb9d7ef86abfc08ae85810435d WatchSource:0}: Error finding container 89022463ae98cdd739b1e811a7cf996695c85ffb9d7ef86abfc08ae85810435d: Status 404 returned error can't find the container with id 89022463ae98cdd739b1e811a7cf996695c85ffb9d7ef86abfc08ae85810435d
Jul 01 17:23:39.676055 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:39.676008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-848c499849-vgmzr" event={"ID":"427e33b1-7ec9-4656-9e2c-a75ae42a44c3","Type":"ContainerStarted","Data":"89022463ae98cdd739b1e811a7cf996695c85ffb9d7ef86abfc08ae85810435d"}
Jul 01 17:23:41.686100 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:41.686012    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-848c499849-vgmzr" event={"ID":"427e33b1-7ec9-4656-9e2c-a75ae42a44c3","Type":"ContainerStarted","Data":"66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6"}
Jul 01 17:23:41.686472 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:41.686156    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:23:41.705385 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:23:41.705337    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-848c499849-vgmzr" podStartSLOduration=1.298343313 podStartE2EDuration="3.70532328s" totalImagesPullingTime="2.406979967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:23:41.703912466 +0000 UTC m=+452.109573243" watchObservedRunningTime="2026-07-01 17:23:41.70532328 +0000 UTC m=+452.110984056"
Jul 01 17:24:12.695313 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:12.695277    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:24:13.823286 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:13.823244    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-848c499849-vgmzr"]
Jul 01 17:24:13.823705 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:13.823482    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-848c499849-vgmzr" podUID="427e33b1-7ec9-4656-9e2c-a75ae42a44c3" containerName="manager" containerID="cri-o://66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6" gracePeriod=10
Jul 01 17:24:14.063998 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.063973    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:24:14.174680 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.174599    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm2wz\" (UniqueName: \"kubernetes.io/projected/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-kube-api-access-wm2wz\") pod \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") "
Jul 01 17:24:14.174680 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.174641    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-cert\") pod \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\" (UID: \"427e33b1-7ec9-4656-9e2c-a75ae42a44c3\") "
Jul 01 17:24:14.176837 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.176806    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-cert" (OuterVolumeSpecName: "cert") pod "427e33b1-7ec9-4656-9e2c-a75ae42a44c3" (UID: "427e33b1-7ec9-4656-9e2c-a75ae42a44c3"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:24:14.176951 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.176810    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-kube-api-access-wm2wz" (OuterVolumeSpecName: "kube-api-access-wm2wz") pod "427e33b1-7ec9-4656-9e2c-a75ae42a44c3" (UID: "427e33b1-7ec9-4656-9e2c-a75ae42a44c3"). InnerVolumeSpecName "kube-api-access-wm2wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:24:14.275948 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.275896    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2wz\" (UniqueName: \"kubernetes.io/projected/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-kube-api-access-wm2wz\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:24:14.275948 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.275945    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/427e33b1-7ec9-4656-9e2c-a75ae42a44c3-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:24:14.804457 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.804420    2576 generic.go:356] "Generic (PLEG): container finished" podID="427e33b1-7ec9-4656-9e2c-a75ae42a44c3" containerID="66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6" exitCode=0
Jul 01 17:24:14.804654 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.804465    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-848c499849-vgmzr"
Jul 01 17:24:14.804654 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.804486    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-848c499849-vgmzr" event={"ID":"427e33b1-7ec9-4656-9e2c-a75ae42a44c3","Type":"ContainerDied","Data":"66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6"}
Jul 01 17:24:14.804654 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.804546    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-848c499849-vgmzr" event={"ID":"427e33b1-7ec9-4656-9e2c-a75ae42a44c3","Type":"ContainerDied","Data":"89022463ae98cdd739b1e811a7cf996695c85ffb9d7ef86abfc08ae85810435d"}
Jul 01 17:24:14.804654 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.804568    2576 scope.go:117] "RemoveContainer" containerID="66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6"
Jul 01 17:24:14.812919 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.812895    2576 scope.go:117] "RemoveContainer" containerID="66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6"
Jul 01 17:24:14.813202 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:24:14.813177    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6\": container with ID starting with 66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6 not found: ID does not exist" containerID="66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6"
Jul 01 17:24:14.813268 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.813211    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6"} err="failed to get container status \"66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6\": rpc error: code = NotFound desc = could not find container \"66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6\": container with ID starting with 66a5db685bb61b7f6a5e7b592fc3ac5bbcfca8b43c6ed3137af3460ed58887c6 not found: ID does not exist"
Jul 01 17:24:14.827329 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.827295    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-848c499849-vgmzr"]
Jul 01 17:24:14.830764 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:14.830736    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-848c499849-vgmzr"]
Jul 01 17:24:16.047241 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:16.047211    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="427e33b1-7ec9-4656-9e2c-a75ae42a44c3" path="/var/lib/kubelet/pods/427e33b1-7ec9-4656-9e2c-a75ae42a44c3/volumes"
Jul 01 17:24:49.586081 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.586045    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-228h5"]
Jul 01 17:24:49.586485 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.586301    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427e33b1-7ec9-4656-9e2c-a75ae42a44c3" containerName="manager"
Jul 01 17:24:49.586485 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.586312    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="427e33b1-7ec9-4656-9e2c-a75ae42a44c3" containerName="manager"
Jul 01 17:24:49.586485 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.586368    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="427e33b1-7ec9-4656-9e2c-a75ae42a44c3" containerName="manager"
Jul 01 17:24:49.589240 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.589215    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.594130 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.594110    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-tf2zj\""
Jul 01 17:24:49.594244 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.594109    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 01 17:24:49.595364 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.595345    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 01 17:24:49.595451 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.595347    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 01 17:24:49.610793 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.610764    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-228h5"]
Jul 01 17:24:49.621723 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.621693    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6xmcq"]
Jul 01 17:24:49.624721 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.624705    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:49.627831 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.627816    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-5n6rc\""
Jul 01 17:24:49.627951 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.627936    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 01 17:24:49.636084 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.636060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6xmcq"]
Jul 01 17:24:49.738615 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.738580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4prs\" (UniqueName: \"kubernetes.io/projected/b6d29a38-1413-448e-b3c8-7c6c94feedf4-kube-api-access-b4prs\") pod \"model-serving-api-fd65d7d6-228h5\" (UID: \"b6d29a38-1413-448e-b3c8-7c6c94feedf4\") " pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.738615 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.738620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0-cert\") pod \"odh-model-controller-84b54b7f97-6xmcq\" (UID: \"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0\") " pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:49.738804 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.738639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d29a38-1413-448e-b3c8-7c6c94feedf4-tls-certs\") pod \"model-serving-api-fd65d7d6-228h5\" (UID: \"b6d29a38-1413-448e-b3c8-7c6c94feedf4\") " pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.738804 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.738696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb64l\" (UniqueName: \"kubernetes.io/projected/dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0-kube-api-access-fb64l\") pod \"odh-model-controller-84b54b7f97-6xmcq\" (UID: \"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0\") " pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:49.839296 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.839213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0-cert\") pod \"odh-model-controller-84b54b7f97-6xmcq\" (UID: \"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0\") " pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:49.839296 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.839250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d29a38-1413-448e-b3c8-7c6c94feedf4-tls-certs\") pod \"model-serving-api-fd65d7d6-228h5\" (UID: \"b6d29a38-1413-448e-b3c8-7c6c94feedf4\") " pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.839296 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.839285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb64l\" (UniqueName: \"kubernetes.io/projected/dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0-kube-api-access-fb64l\") pod \"odh-model-controller-84b54b7f97-6xmcq\" (UID: \"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0\") " pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:49.839601 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.839359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4prs\" (UniqueName: \"kubernetes.io/projected/b6d29a38-1413-448e-b3c8-7c6c94feedf4-kube-api-access-b4prs\") pod \"model-serving-api-fd65d7d6-228h5\" (UID: \"b6d29a38-1413-448e-b3c8-7c6c94feedf4\") " pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.841873 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.841842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0-cert\") pod \"odh-model-controller-84b54b7f97-6xmcq\" (UID: \"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0\") " pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:49.841974 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.841842    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d29a38-1413-448e-b3c8-7c6c94feedf4-tls-certs\") pod \"model-serving-api-fd65d7d6-228h5\" (UID: \"b6d29a38-1413-448e-b3c8-7c6c94feedf4\") " pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.853454 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.853428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4prs\" (UniqueName: \"kubernetes.io/projected/b6d29a38-1413-448e-b3c8-7c6c94feedf4-kube-api-access-b4prs\") pod \"model-serving-api-fd65d7d6-228h5\" (UID: \"b6d29a38-1413-448e-b3c8-7c6c94feedf4\") " pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.855975 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.855953    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb64l\" (UniqueName: \"kubernetes.io/projected/dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0-kube-api-access-fb64l\") pod \"odh-model-controller-84b54b7f97-6xmcq\" (UID: \"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0\") " pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:49.899009 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.898980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:49.935572 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:49.935512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:50.036830 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:50.036790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-228h5"]
Jul 01 17:24:50.039988 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:24:50.039961    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d29a38_1413_448e_b3c8_7c6c94feedf4.slice/crio-593d4a7c3609f4122e1579d64db1d9bd3001f17b5abaa0726ff738b6cdff1ad4 WatchSource:0}: Error finding container 593d4a7c3609f4122e1579d64db1d9bd3001f17b5abaa0726ff738b6cdff1ad4: Status 404 returned error can't find the container with id 593d4a7c3609f4122e1579d64db1d9bd3001f17b5abaa0726ff738b6cdff1ad4
Jul 01 17:24:50.081026 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:50.080888    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-6xmcq"]
Jul 01 17:24:50.083563 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:24:50.083534    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2c74f9_bb34_4173_a7cd_826e7e8bf8a0.slice/crio-2a88dee532465d692af7865e79b0c10d8df6513209258fdac30ea78123004ff6 WatchSource:0}: Error finding container 2a88dee532465d692af7865e79b0c10d8df6513209258fdac30ea78123004ff6: Status 404 returned error can't find the container with id 2a88dee532465d692af7865e79b0c10d8df6513209258fdac30ea78123004ff6
Jul 01 17:24:50.934565 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:50.934502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-6xmcq" event={"ID":"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0","Type":"ContainerStarted","Data":"2a88dee532465d692af7865e79b0c10d8df6513209258fdac30ea78123004ff6"}
Jul 01 17:24:50.936014 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:50.935986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-228h5" event={"ID":"b6d29a38-1413-448e-b3c8-7c6c94feedf4","Type":"ContainerStarted","Data":"593d4a7c3609f4122e1579d64db1d9bd3001f17b5abaa0726ff738b6cdff1ad4"}
Jul 01 17:24:53.952192 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:53.952087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-6xmcq" event={"ID":"dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0","Type":"ContainerStarted","Data":"aef9cd663b73566f74fa10da055cedc64665c91a78a29251f74a6c7d66b77fad"}
Jul 01 17:24:53.952192 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:53.952156    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:24:53.954012 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:53.953985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-228h5" event={"ID":"b6d29a38-1413-448e-b3c8-7c6c94feedf4","Type":"ContainerStarted","Data":"df9b4c1b73bf0a23081d1f3a4a6c5ea6af23b9c44bc1481a46a4978a92e90379"}
Jul 01 17:24:53.954141 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:53.954125    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:24:53.972874 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:53.972831    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-6xmcq" podStartSLOduration=1.3828073029999999 podStartE2EDuration="4.972817858s" totalImagesPullingTime="3.590010555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:24:53.972355248 +0000 UTC m=+524.378016025" watchObservedRunningTime="2026-07-01 17:24:53.972817858 +0000 UTC m=+524.378478633"
Jul 01 17:24:53.996359 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:24:53.996305    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-228h5" podStartSLOduration=1.367664105 podStartE2EDuration="4.996290766s" totalImagesPullingTime="3.628626661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:24:53.995245745 +0000 UTC m=+524.400906536" watchObservedRunningTime="2026-07-01 17:24:53.996290766 +0000 UTC m=+524.401951542"
Jul 01 17:25:04.960048 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:25:04.959964    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-6xmcq"
Jul 01 17:25:04.962044 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:25:04.962022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-228h5"
Jul 01 17:26:09.993615 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:09.993587    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:26:09.994491 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:09.994473    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:26:37.757093 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.757015    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"]
Jul 01 17:26:37.759819 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.759800    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:37.762825 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.762668    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:26:37.762825 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.762708    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0f1eb-predictor-serving-cert\""
Jul 01 17:26:37.762825 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.762735    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:26:37.762825 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.762668    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:26:37.762825 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.762667    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0f1eb-kube-rbac-proxy-sar-config\""
Jul 01 17:26:37.763172 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.762849    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:26:37.763172 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.763126    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xldfq\""
Jul 01 17:26:37.772752 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.772723    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"]
Jul 01 17:26:37.930445 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.930403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2999af9d-f5d1-4b86-8417-524f940f139d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:37.930645 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.930467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:37.930645 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.930559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:37.930645 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.930580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg559\" (UniqueName: \"kubernetes.io/projected/2999af9d-f5d1-4b86-8417-524f940f139d-kube-api-access-tg559\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:37.930645 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:37.930601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2999af9d-f5d1-4b86-8417-524f940f139d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.031775 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.031690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2999af9d-f5d1-4b86-8417-524f940f139d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.031775 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.031749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.032005 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.031790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.032005 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.031815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg559\" (UniqueName: \"kubernetes.io/projected/2999af9d-f5d1-4b86-8417-524f940f139d-kube-api-access-tg559\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.032005 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.031836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2999af9d-f5d1-4b86-8417-524f940f139d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.032505 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.032474    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.032505 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.032494    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2999af9d-f5d1-4b86-8417-524f940f139d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.032695 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.032616    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.034429 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.034407    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2999af9d-f5d1-4b86-8417-524f940f139d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.040830 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.040810    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg559\" (UniqueName: \"kubernetes.io/projected/2999af9d-f5d1-4b86-8417-524f940f139d-kube-api-access-tg559\") pod \"isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.070402 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.070375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:26:38.200178 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.200156    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"]
Jul 01 17:26:38.202471 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:26:38.202447    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2999af9d_f5d1_4b86_8417_524f940f139d.slice/crio-793f01b57f3894120a8a9cddaf9022ceebddc17f43962ac5d36d0b29ccb2fb99 WatchSource:0}: Error finding container 793f01b57f3894120a8a9cddaf9022ceebddc17f43962ac5d36d0b29ccb2fb99: Status 404 returned error can't find the container with id 793f01b57f3894120a8a9cddaf9022ceebddc17f43962ac5d36d0b29ccb2fb99
Jul 01 17:26:38.350610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:38.350500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerStarted","Data":"793f01b57f3894120a8a9cddaf9022ceebddc17f43962ac5d36d0b29ccb2fb99"}
Jul 01 17:26:42.367766 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:42.367725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerStarted","Data":"7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1"}
Jul 01 17:26:46.384036 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:46.384002    2576 generic.go:356] "Generic (PLEG): container finished" podID="2999af9d-f5d1-4b86-8417-524f940f139d" containerID="7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1" exitCode=0
Jul 01 17:26:46.384395 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:46.384059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerDied","Data":"7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1"}
Jul 01 17:26:59.897388 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:26:59.897369    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:27:00.457938 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:00.457900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerStarted","Data":"ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c"}
Jul 01 17:27:03.472866 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:03.472832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerStarted","Data":"48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26"}
Jul 01 17:27:06.486964 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:06.486928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerStarted","Data":"265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478"}
Jul 01 17:27:06.487407 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:06.487024    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:27:06.487407 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:06.487176    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:27:06.488508 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:06.488482    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:06.509115 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:06.509070    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podStartSLOduration=2.74321494 podStartE2EDuration="29.509057792s" totalImagesPullingTime="23.15353495s" totalInitContainerRuntime="3.612307902s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:26:37 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:27:06.507450331 +0000 UTC m=+656.913111108" watchObservedRunningTime="2026-07-01 17:27:06.509057792 +0000 UTC m=+656.914718568"
Jul 01 17:27:07.491250 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:07.491209    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:27:07.491745 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:07.491209    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:07.492421 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:07.492392    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:27:07.495242 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:07.495223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:27:08.494684 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:08.494648    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:08.495131 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:08.494963    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:27:09.498216 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:09.498161    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:09.498646 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:09.498494    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:27:19.498493 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:19.498446    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:19.499066 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:19.498892    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:27:29.499062 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:29.498998    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:29.499587 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:29.499563    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:27:39.498812 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:39.498760    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:39.499284 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:39.499193    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:27:49.498181 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:49.498127    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:49.498610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:49.498581    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:27:59.498467 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:59.498418    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:27:59.498936 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:27:59.498914    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:28:09.498513 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:09.498400    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:28:09.498949 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:09.498918    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:28:19.499129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:19.499095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:28:19.499737 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:19.499318    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:28:32.900810 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.900775    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"]
Jul 01 17:28:32.904330 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.904308    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:32.906973 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.906950    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-99462-predictor-serving-cert\""
Jul 01 17:28:32.907119 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.907096    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-99462-kube-rbac-proxy-sar-config\""
Jul 01 17:28:32.917103 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.917078    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"]
Jul 01 17:28:32.931688 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.931655    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"]
Jul 01 17:28:32.931982 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.931932    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" containerID="cri-o://ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c" gracePeriod=30
Jul 01 17:28:32.932199 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.931967    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" containerID="cri-o://48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26" gracePeriod=30
Jul 01 17:28:32.932310 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.931992    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" containerID="cri-o://265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478" gracePeriod=30
Jul 01 17:28:32.963065 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.963026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5ww\" (UniqueName: \"kubernetes.io/projected/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-api-access-wc5ww\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:32.963065 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.963068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-cabundle-cert\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:32.963292 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.963094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:32.963292 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.963127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:32.963292 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.963157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:32.974715 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.974679    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"]
Jul 01 17:28:32.978211 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.978180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:32.980897 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.980879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-99462-predictor-serving-cert\""
Jul 01 17:28:32.981020 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.980919    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-99462-kube-rbac-proxy-sar-config\""
Jul 01 17:28:32.989650 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:32.989626    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"]
Jul 01 17:28:33.064165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.064165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.064384 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7jgp\" (UniqueName: \"kubernetes.io/projected/b8581e75-339c-41ac-9673-19613153be80-kube-api-access-z7jgp\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.064384 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8581e75-339c-41ac-9673-19613153be80-proxy-tls\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.064384 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.064384 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.064627 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:28:33.064441    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-serving-cert: secret "isvc-sklearn-graph-raw-99462-predictor-serving-cert" not found
Jul 01 17:28:33.064627 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-cabundle-cert\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.064627 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8581e75-339c-41ac-9673-19613153be80-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.064627 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:28:33.064568    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls podName:21d1992b-eeb2-4852-a0b9-6f6587de84ad nodeName:}" failed. No retries permitted until 2026-07-01 17:28:33.564543844 +0000 UTC m=+743.970204597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls") pod "isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" (UID: "21d1992b-eeb2-4852-a0b9-6f6587de84ad") : secret "isvc-sklearn-graph-raw-99462-predictor-serving-cert" not found
Jul 01 17:28:33.064627 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064571    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.064952 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5ww\" (UniqueName: \"kubernetes.io/projected/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-api-access-wc5ww\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.064952 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.064659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-cabundle-cert\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.065115 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.065092    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.065166 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.065152    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-cabundle-cert\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.073930 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.073900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5ww\" (UniqueName: \"kubernetes.io/projected/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-api-access-wc5ww\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.165509 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.165416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-cabundle-cert\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.165509 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.165464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8581e75-339c-41ac-9673-19613153be80-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.165759 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.165565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7jgp\" (UniqueName: \"kubernetes.io/projected/b8581e75-339c-41ac-9673-19613153be80-kube-api-access-z7jgp\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.165759 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.165605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8581e75-339c-41ac-9673-19613153be80-proxy-tls\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.165759 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.165658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.165969 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.165941    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8581e75-339c-41ac-9673-19613153be80-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.166177 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.166154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-cabundle-cert\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.166315 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.166287    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.168030 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.168012    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8581e75-339c-41ac-9673-19613153be80-proxy-tls\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.176442 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.176420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7jgp\" (UniqueName: \"kubernetes.io/projected/b8581e75-339c-41ac-9673-19613153be80-kube-api-access-z7jgp\") pod \"isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.289181 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.289135    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:28:33.422997 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.422969    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"]
Jul 01 17:28:33.425156 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:28:33.425133    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8581e75_339c_41ac_9673_19613153be80.slice/crio-654b30533e6a9a4915e3c239376bfc56eb95ac39ea5529b3a497b69ebb6c0705 WatchSource:0}: Error finding container 654b30533e6a9a4915e3c239376bfc56eb95ac39ea5529b3a497b69ebb6c0705: Status 404 returned error can't find the container with id 654b30533e6a9a4915e3c239376bfc56eb95ac39ea5529b3a497b69ebb6c0705
Jul 01 17:28:33.569915 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.569866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.572238 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.572215    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls\") pod \"isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.816073 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.816041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:33.830079 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.830043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerStarted","Data":"659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41"}
Jul 01 17:28:33.830237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.830124    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerStarted","Data":"654b30533e6a9a4915e3c239376bfc56eb95ac39ea5529b3a497b69ebb6c0705"}
Jul 01 17:28:33.832871 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.832841    2576 generic.go:356] "Generic (PLEG): container finished" podID="2999af9d-f5d1-4b86-8417-524f940f139d" containerID="48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26" exitCode=2
Jul 01 17:28:33.832979 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.832913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerDied","Data":"48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26"}
Jul 01 17:28:33.956647 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:33.956621    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"]
Jul 01 17:28:33.959465 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:28:33.959439    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d1992b_eeb2_4852_a0b9_6f6587de84ad.slice/crio-ac63f6588083c3125550432e1b332cb19eea7ddc6320ef770cd5aad823f33af6 WatchSource:0}: Error finding container ac63f6588083c3125550432e1b332cb19eea7ddc6320ef770cd5aad823f33af6: Status 404 returned error can't find the container with id ac63f6588083c3125550432e1b332cb19eea7ddc6320ef770cd5aad823f33af6
Jul 01 17:28:34.845010 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:34.844968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerStarted","Data":"8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f"}
Jul 01 17:28:34.845312 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:34.845076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerStarted","Data":"ac63f6588083c3125550432e1b332cb19eea7ddc6320ef770cd5aad823f33af6"}
Jul 01 17:28:37.491736 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:37.491700    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 17:28:37.860318 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:37.860290    2576 generic.go:356] "Generic (PLEG): container finished" podID="b8581e75-339c-41ac-9673-19613153be80" containerID="659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41" exitCode=0
Jul 01 17:28:37.860460 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:37.860361    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerDied","Data":"659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41"}
Jul 01 17:28:37.861927 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:37.861903    2576 generic.go:356] "Generic (PLEG): container finished" podID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerID="8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f" exitCode=0
Jul 01 17:28:37.862028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:37.861983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerDied","Data":"8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f"}
Jul 01 17:28:37.864443 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:37.864423    2576 generic.go:356] "Generic (PLEG): container finished" podID="2999af9d-f5d1-4b86-8417-524f940f139d" containerID="ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c" exitCode=0
Jul 01 17:28:37.864621 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:37.864487    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerDied","Data":"ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c"}
Jul 01 17:28:38.875300 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:38.875251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerStarted","Data":"6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777"}
Jul 01 17:28:38.875773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:38.875330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerStarted","Data":"d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef"}
Jul 01 17:28:38.875773 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:38.875584    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:38.897347 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:38.897292    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podStartSLOduration=3.082475471 podStartE2EDuration="6.897273881s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81479841s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:28:38.895080057 +0000 UTC m=+749.300740872" watchObservedRunningTime="2026-07-01 17:28:38.897273881 +0000 UTC m=+749.302934656"
Jul 01 17:28:39.498784 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:39.498734    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:28:39.499064 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:39.499030    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:28:39.882011 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:39.881925    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:39.883425 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:39.883393    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:28:40.886340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:40.886287    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:28:42.492035 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:42.491995    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 17:28:45.891436 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:45.891408    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:28:45.891938 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:45.891907    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:28:47.492266 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:47.492222    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 17:28:47.492728 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:47.492404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:28:49.498745 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:49.498696    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:28:49.499156 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:49.499071    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:28:52.492190 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:52.492145    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 17:28:55.892033 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:55.891991    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:28:57.492201 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:57.492161    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 17:28:59.498613 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:59.498565    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:28:59.499104 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:59.498719    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:28:59.499104 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:59.498977    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:28:59.499104 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:28:59.499075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:29:02.491932 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:02.491881    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 17:29:05.892181 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:05.892141    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:07.492078 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:07.492031    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 01 17:29:09.498263 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.498201    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Jul 01 17:29:09.498792 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.498215    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent" probeResult="failure" output="Get \"http://10.132.0.21:9081/\": dial tcp 10.132.0.21:9081: connect: connection refused"
Jul 01 17:29:09.766477 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.766442    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:29:09.875997 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.875962    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2999af9d-f5d1-4b86-8417-524f940f139d-proxy-tls\") pod \"2999af9d-f5d1-4b86-8417-524f940f139d\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") "
Jul 01 17:29:09.876166 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.876033    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg559\" (UniqueName: \"kubernetes.io/projected/2999af9d-f5d1-4b86-8417-524f940f139d-kube-api-access-tg559\") pod \"2999af9d-f5d1-4b86-8417-524f940f139d\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") "
Jul 01 17:29:09.876166 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.876066    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-kube-rbac-proxy-sar-config\") pod \"2999af9d-f5d1-4b86-8417-524f940f139d\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") "
Jul 01 17:29:09.876166 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.876090    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-cabundle-cert\") pod \"2999af9d-f5d1-4b86-8417-524f940f139d\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") "
Jul 01 17:29:09.876166 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.876128    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2999af9d-f5d1-4b86-8417-524f940f139d-kserve-provision-location\") pod \"2999af9d-f5d1-4b86-8417-524f940f139d\" (UID: \"2999af9d-f5d1-4b86-8417-524f940f139d\") "
Jul 01 17:29:09.876504 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.876469    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2999af9d-f5d1-4b86-8417-524f940f139d" (UID: "2999af9d-f5d1-4b86-8417-524f940f139d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:29:09.876629 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.876473    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2999af9d-f5d1-4b86-8417-524f940f139d" (UID: "2999af9d-f5d1-4b86-8417-524f940f139d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:29:09.876629 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.876510    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2999af9d-f5d1-4b86-8417-524f940f139d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2999af9d-f5d1-4b86-8417-524f940f139d" (UID: "2999af9d-f5d1-4b86-8417-524f940f139d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:29:09.878293 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.878264    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2999af9d-f5d1-4b86-8417-524f940f139d-kube-api-access-tg559" (OuterVolumeSpecName: "kube-api-access-tg559") pod "2999af9d-f5d1-4b86-8417-524f940f139d" (UID: "2999af9d-f5d1-4b86-8417-524f940f139d"). InnerVolumeSpecName "kube-api-access-tg559". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:29:09.878383 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.878305    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2999af9d-f5d1-4b86-8417-524f940f139d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2999af9d-f5d1-4b86-8417-524f940f139d" (UID: "2999af9d-f5d1-4b86-8417-524f940f139d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:29:09.977591 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.977512    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg559\" (UniqueName: \"kubernetes.io/projected/2999af9d-f5d1-4b86-8417-524f940f139d-kube-api-access-tg559\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:29:09.977591 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.977548    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:29:09.977591 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.977559    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2999af9d-f5d1-4b86-8417-524f940f139d-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:29:09.977591 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.977568    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2999af9d-f5d1-4b86-8417-524f940f139d-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:29:09.977591 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:09.977576    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2999af9d-f5d1-4b86-8417-524f940f139d-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:29:10.017818 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.017776    2576 generic.go:356] "Generic (PLEG): container finished" podID="2999af9d-f5d1-4b86-8417-524f940f139d" containerID="265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478" exitCode=0
Jul 01 17:29:10.017985 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.017837    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"
Jul 01 17:29:10.017985 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.017839    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerDied","Data":"265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478"}
Jul 01 17:29:10.017985 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.017934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj" event={"ID":"2999af9d-f5d1-4b86-8417-524f940f139d","Type":"ContainerDied","Data":"793f01b57f3894120a8a9cddaf9022ceebddc17f43962ac5d36d0b29ccb2fb99"}
Jul 01 17:29:10.017985 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.017964    2576 scope.go:117] "RemoveContainer" containerID="265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478"
Jul 01 17:29:10.027659 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.027444    2576 scope.go:117] "RemoveContainer" containerID="48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26"
Jul 01 17:29:10.035965 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.035946    2576 scope.go:117] "RemoveContainer" containerID="ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c"
Jul 01 17:29:10.043787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.043763    2576 scope.go:117] "RemoveContainer" containerID="7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1"
Jul 01 17:29:10.048698 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.048676    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"]
Jul 01 17:29:10.048893 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.048874    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0f1eb-predictor-799bfb48b8-2njmj"]
Jul 01 17:29:10.051916 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.051897    2576 scope.go:117] "RemoveContainer" containerID="265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478"
Jul 01 17:29:10.052184 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:29:10.052162    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478\": container with ID starting with 265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478 not found: ID does not exist" containerID="265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478"
Jul 01 17:29:10.052252 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.052191    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478"} err="failed to get container status \"265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478\": rpc error: code = NotFound desc = could not find container \"265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478\": container with ID starting with 265863db326800999f8c8d7ef8499745072c15910c64af1536ee79b9f4bb5478 not found: ID does not exist"
Jul 01 17:29:10.052252 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.052210    2576 scope.go:117] "RemoveContainer" containerID="48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26"
Jul 01 17:29:10.052458 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:29:10.052442    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26\": container with ID starting with 48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26 not found: ID does not exist" containerID="48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26"
Jul 01 17:29:10.052497 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.052462    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26"} err="failed to get container status \"48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26\": rpc error: code = NotFound desc = could not find container \"48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26\": container with ID starting with 48be4f5f21aa8e7b912a871bdb8b88207473bb4f2f2d8f40fac22d8dd408ce26 not found: ID does not exist"
Jul 01 17:29:10.052497 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.052476    2576 scope.go:117] "RemoveContainer" containerID="ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c"
Jul 01 17:29:10.052741 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:29:10.052723    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c\": container with ID starting with ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c not found: ID does not exist" containerID="ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c"
Jul 01 17:29:10.052788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.052747    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c"} err="failed to get container status \"ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c\": rpc error: code = NotFound desc = could not find container \"ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c\": container with ID starting with ff5118e2aad9ea6bed8e15b5e29cf87bc015b913fd4ae04d13a1f69afffa259c not found: ID does not exist"
Jul 01 17:29:10.052788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.052762    2576 scope.go:117] "RemoveContainer" containerID="7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1"
Jul 01 17:29:10.052970 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:29:10.052953    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1\": container with ID starting with 7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1 not found: ID does not exist" containerID="7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1"
Jul 01 17:29:10.053025 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:10.052977    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1"} err="failed to get container status \"7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1\": rpc error: code = NotFound desc = could not find container \"7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1\": container with ID starting with 7ebfbc1fcba97dbd490c990535d1fcccdcdb002d601c940f2e07e3572844edd1 not found: ID does not exist"
Jul 01 17:29:11.023943 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:11.023900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerStarted","Data":"6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024"}
Jul 01 17:29:11.024427 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:11.023978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerStarted","Data":"7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434"}
Jul 01 17:29:11.024427 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:11.024312    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:29:11.024589 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:11.024555    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:29:11.026027 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:11.025995    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:29:11.044378 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:11.044332    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podStartSLOduration=2.949685063 podStartE2EDuration="39.044315465s" totalImagesPullingTime="32.174955419s" totalInitContainerRuntime="3.919674983s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:29:11.043041289 +0000 UTC m=+781.448702101" watchObservedRunningTime="2026-07-01 17:29:11.044315465 +0000 UTC m=+781.449976291"
Jul 01 17:29:12.029458 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:12.029410    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:29:12.047897 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:12.047864    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" path="/var/lib/kubelet/pods/2999af9d-f5d1-4b86-8417-524f940f139d/volumes"
Jul 01 17:29:15.891905 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:15.891866    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:17.034215 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:17.034186    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:29:17.034680 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:17.034653    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:29:25.892776 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:25.892736    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:27.035661 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:27.035619    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:29:35.892077 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:35.891994    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:37.034867 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:37.034823    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:29:45.892272 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:45.892233    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:29:47.034957 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:47.034918    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:29:55.893278 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:55.893244    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:29:57.035452 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:29:57.035411    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:30:07.035238 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:07.035196    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 01 17:30:17.035689 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:17.035657    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:30:33.190436 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190404    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"]
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190744    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container"
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190760    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container"
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190774    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="storage-initializer"
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190779    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="storage-initializer"
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190792    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy"
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190798    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy"
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190805    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent"
Jul 01 17:30:33.190807 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190811    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent"
Jul 01 17:30:33.191092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190862    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="agent"
Jul 01 17:30:33.191092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190870    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kserve-container"
Jul 01 17:30:33.191092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.190877    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2999af9d-f5d1-4b86-8417-524f940f139d" containerName="kube-rbac-proxy"
Jul 01 17:30:33.193835 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.193812    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.196809 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.196785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-bca43-predictor-serving-cert\""
Jul 01 17:30:33.197053 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.196813    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-bca43-kube-rbac-proxy-sar-config\""
Jul 01 17:30:33.207157 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.207133    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"]
Jul 01 17:30:33.207466 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.207443    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kube-rbac-proxy" containerID="cri-o://6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777" gracePeriod=30
Jul 01 17:30:33.207610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.207582    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" containerID="cri-o://d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef" gracePeriod=30
Jul 01 17:30:33.211048 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.211028    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"]
Jul 01 17:30:33.259790 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.259754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68bk\" (UniqueName: \"kubernetes.io/projected/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-api-access-l68bk\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.259959 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.259816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.259959 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.259858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/369fd6fa-217d-448b-94a2-bf0938246ab5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.259959 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.259887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.259959 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.259943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.336099 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.336068    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"]
Jul 01 17:30:33.336342 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.336302    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container" containerID="cri-o://7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434" gracePeriod=30
Jul 01 17:30:33.336440 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.336332    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kube-rbac-proxy" containerID="cri-o://6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024" gracePeriod=30
Jul 01 17:30:33.360442 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.360410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68bk\" (UniqueName: \"kubernetes.io/projected/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-api-access-l68bk\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.361028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.360465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.361028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.360492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/369fd6fa-217d-448b-94a2-bf0938246ab5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.361028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.360558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.361028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.360623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.361028 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:33.360831    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-bca43-predictor-serving-cert" not found
Jul 01 17:30:33.361028 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:33.360924    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls podName:369fd6fa-217d-448b-94a2-bf0938246ab5 nodeName:}" failed. No retries permitted until 2026-07-01 17:30:33.860903055 +0000 UTC m=+864.266563812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" (UID: "369fd6fa-217d-448b-94a2-bf0938246ab5") : secret "isvc-sklearn-graph-raw-hpa-bca43-predictor-serving-cert" not found
Jul 01 17:30:33.361028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.360948    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/369fd6fa-217d-448b-94a2-bf0938246ab5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.361291 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.361270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.361329 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.361314    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.369582 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.369548    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68bk\" (UniqueName: \"kubernetes.io/projected/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-api-access-l68bk\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.373396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.373374    2576 generic.go:356] "Generic (PLEG): container finished" podID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerID="6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777" exitCode=2
Jul 01 17:30:33.373494 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.373446    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerDied","Data":"6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777"}
Jul 01 17:30:33.864386 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.864346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:33.866770 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:33.866746    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:34.106832 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:34.106795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:34.238987 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:34.238760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"]
Jul 01 17:30:34.241670 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:30:34.241639    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369fd6fa_217d_448b_94a2_bf0938246ab5.slice/crio-08ddaae7f67df085b1da3f0f3f62b219c5aa75f359e988bfae82a2aba0bac5ee WatchSource:0}: Error finding container 08ddaae7f67df085b1da3f0f3f62b219c5aa75f359e988bfae82a2aba0bac5ee: Status 404 returned error can't find the container with id 08ddaae7f67df085b1da3f0f3f62b219c5aa75f359e988bfae82a2aba0bac5ee
Jul 01 17:30:34.379809 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:34.379730    2576 generic.go:356] "Generic (PLEG): container finished" podID="b8581e75-339c-41ac-9673-19613153be80" containerID="6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024" exitCode=2
Jul 01 17:30:34.379953 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:34.379810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerDied","Data":"6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024"}
Jul 01 17:30:34.381368 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:34.381346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerStarted","Data":"9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0"}
Jul 01 17:30:34.381487 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:34.381389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerStarted","Data":"08ddaae7f67df085b1da3f0f3f62b219c5aa75f359e988bfae82a2aba0bac5ee"}
Jul 01 17:30:35.886932 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:35.886886    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 01 17:30:35.892598 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:35.892567    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 01 17:30:36.881585 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.881562    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:30:36.990175 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990080    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8581e75-339c-41ac-9673-19613153be80-kserve-provision-location\") pod \"b8581e75-339c-41ac-9673-19613153be80\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") "
Jul 01 17:30:36.990175 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990124    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8581e75-339c-41ac-9673-19613153be80-proxy-tls\") pod \"b8581e75-339c-41ac-9673-19613153be80\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") "
Jul 01 17:30:36.990175 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990147    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-cabundle-cert\") pod \"b8581e75-339c-41ac-9673-19613153be80\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") "
Jul 01 17:30:36.990641 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990181    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-kube-rbac-proxy-sar-config\") pod \"b8581e75-339c-41ac-9673-19613153be80\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") "
Jul 01 17:30:36.990641 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990201    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7jgp\" (UniqueName: \"kubernetes.io/projected/b8581e75-339c-41ac-9673-19613153be80-kube-api-access-z7jgp\") pod \"b8581e75-339c-41ac-9673-19613153be80\" (UID: \"b8581e75-339c-41ac-9673-19613153be80\") "
Jul 01 17:30:36.990641 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990484    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8581e75-339c-41ac-9673-19613153be80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8581e75-339c-41ac-9673-19613153be80" (UID: "b8581e75-339c-41ac-9673-19613153be80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:30:36.990641 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990539    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8581e75-339c-41ac-9673-19613153be80" (UID: "b8581e75-339c-41ac-9673-19613153be80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:30:36.990641 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.990595    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8581e75-339c-41ac-9673-19613153be80" (UID: "b8581e75-339c-41ac-9673-19613153be80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:30:36.992343 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.992319    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8581e75-339c-41ac-9673-19613153be80-kube-api-access-z7jgp" (OuterVolumeSpecName: "kube-api-access-z7jgp") pod "b8581e75-339c-41ac-9673-19613153be80" (UID: "b8581e75-339c-41ac-9673-19613153be80"). InnerVolumeSpecName "kube-api-access-z7jgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:30:36.992395 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:36.992348    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8581e75-339c-41ac-9673-19613153be80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8581e75-339c-41ac-9673-19613153be80" (UID: "b8581e75-339c-41ac-9673-19613153be80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:30:37.091185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.091139    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8581e75-339c-41ac-9673-19613153be80-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.091185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.091181    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8581e75-339c-41ac-9673-19613153be80-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.091185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.091193    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.091185 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.091202    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8581e75-339c-41ac-9673-19613153be80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.091444 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.091211    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7jgp\" (UniqueName: \"kubernetes.io/projected/b8581e75-339c-41ac-9673-19613153be80-kube-api-access-z7jgp\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.395928 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.395842    2576 generic.go:356] "Generic (PLEG): container finished" podID="b8581e75-339c-41ac-9673-19613153be80" containerID="7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434" exitCode=0
Jul 01 17:30:37.395928 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.395898    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"
Jul 01 17:30:37.396106 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.395933    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerDied","Data":"7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434"}
Jul 01 17:30:37.396106 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.396002    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm" event={"ID":"b8581e75-339c-41ac-9673-19613153be80","Type":"ContainerDied","Data":"654b30533e6a9a4915e3c239376bfc56eb95ac39ea5529b3a497b69ebb6c0705"}
Jul 01 17:30:37.396106 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.396039    2576 scope.go:117] "RemoveContainer" containerID="6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024"
Jul 01 17:30:37.404411 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.404393    2576 scope.go:117] "RemoveContainer" containerID="7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434"
Jul 01 17:30:37.411791 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.411770    2576 scope.go:117] "RemoveContainer" containerID="659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41"
Jul 01 17:30:37.418421 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.418396    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"]
Jul 01 17:30:37.419169 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.419151    2576 scope.go:117] "RemoveContainer" containerID="6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024"
Jul 01 17:30:37.419421 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:37.419402    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024\": container with ID starting with 6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024 not found: ID does not exist" containerID="6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024"
Jul 01 17:30:37.419463 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.419431    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024"} err="failed to get container status \"6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024\": rpc error: code = NotFound desc = could not find container \"6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024\": container with ID starting with 6cd8adf8e5c946999c842f8177722816f1632f8ee6262c3ce264f3afa6248024 not found: ID does not exist"
Jul 01 17:30:37.419463 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.419448    2576 scope.go:117] "RemoveContainer" containerID="7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434"
Jul 01 17:30:37.419694 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:37.419677    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434\": container with ID starting with 7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434 not found: ID does not exist" containerID="7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434"
Jul 01 17:30:37.419759 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.419699    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434"} err="failed to get container status \"7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434\": rpc error: code = NotFound desc = could not find container \"7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434\": container with ID starting with 7f2c016baf462729f647b51b6e9c291aec8cb4c5132df95bfa18ce441d789434 not found: ID does not exist"
Jul 01 17:30:37.419759 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.419714    2576 scope.go:117] "RemoveContainer" containerID="659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41"
Jul 01 17:30:37.419889 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:37.419875    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41\": container with ID starting with 659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41 not found: ID does not exist" containerID="659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41"
Jul 01 17:30:37.419927 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.419893    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41"} err="failed to get container status \"659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41\": rpc error: code = NotFound desc = could not find container \"659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41\": container with ID starting with 659e882f3d6e4216b1921f06634d937e4e4bb04caa4b47f2d8c1d7cbd312fa41 not found: ID does not exist"
Jul 01 17:30:37.422438 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.422418    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-99462-predictor-798dbc6969-rngdm"]
Jul 01 17:30:37.853790 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.853766    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:30:37.897961 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.897927    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-rbac-proxy-sar-config\") pod \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") "
Jul 01 17:30:37.898123 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.897994    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wc5ww\" (UniqueName: \"kubernetes.io/projected/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-api-access-wc5ww\") pod \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") "
Jul 01 17:30:37.898123 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.898065    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kserve-provision-location\") pod \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") "
Jul 01 17:30:37.898123 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.898118    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-cabundle-cert\") pod \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") "
Jul 01 17:30:37.898285 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.898145    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls\") pod \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\" (UID: \"21d1992b-eeb2-4852-a0b9-6f6587de84ad\") "
Jul 01 17:30:37.898429 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.898399    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21d1992b-eeb2-4852-a0b9-6f6587de84ad" (UID: "21d1992b-eeb2-4852-a0b9-6f6587de84ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:30:37.898644 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.898431    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21d1992b-eeb2-4852-a0b9-6f6587de84ad" (UID: "21d1992b-eeb2-4852-a0b9-6f6587de84ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:30:37.898644 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.898502    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "21d1992b-eeb2-4852-a0b9-6f6587de84ad" (UID: "21d1992b-eeb2-4852-a0b9-6f6587de84ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:30:37.900197 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.900172    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-api-access-wc5ww" (OuterVolumeSpecName: "kube-api-access-wc5ww") pod "21d1992b-eeb2-4852-a0b9-6f6587de84ad" (UID: "21d1992b-eeb2-4852-a0b9-6f6587de84ad"). InnerVolumeSpecName "kube-api-access-wc5ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:30:37.900197 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.900185    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21d1992b-eeb2-4852-a0b9-6f6587de84ad" (UID: "21d1992b-eeb2-4852-a0b9-6f6587de84ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:30:37.999187 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.999096    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.999187 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.999131    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21d1992b-eeb2-4852-a0b9-6f6587de84ad-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.999187 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.999143    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.999187 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.999154    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc5ww\" (UniqueName: \"kubernetes.io/projected/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kube-api-access-wc5ww\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:37.999187 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:37.999163    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21d1992b-eeb2-4852-a0b9-6f6587de84ad-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:30:38.048084 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.048047    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8581e75-339c-41ac-9673-19613153be80" path="/var/lib/kubelet/pods/b8581e75-339c-41ac-9673-19613153be80/volumes"
Jul 01 17:30:38.402918 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.402828    2576 generic.go:356] "Generic (PLEG): container finished" podID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerID="d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef" exitCode=0
Jul 01 17:30:38.402918 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.402883    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"
Jul 01 17:30:38.402918 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.402906    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerDied","Data":"d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef"}
Jul 01 17:30:38.403129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.402965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm" event={"ID":"21d1992b-eeb2-4852-a0b9-6f6587de84ad","Type":"ContainerDied","Data":"ac63f6588083c3125550432e1b332cb19eea7ddc6320ef770cd5aad823f33af6"}
Jul 01 17:30:38.403129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.402989    2576 scope.go:117] "RemoveContainer" containerID="6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777"
Jul 01 17:30:38.404537 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.404500    2576 generic.go:356] "Generic (PLEG): container finished" podID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerID="9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0" exitCode=0
Jul 01 17:30:38.404637 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.404556    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerDied","Data":"9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0"}
Jul 01 17:30:38.411160 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.411142    2576 scope.go:117] "RemoveContainer" containerID="d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef"
Jul 01 17:30:38.418818 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.418795    2576 scope.go:117] "RemoveContainer" containerID="8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f"
Jul 01 17:30:38.423045 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.423025    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"]
Jul 01 17:30:38.426357 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.426334    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-99462-predictor-5858855d8d-kkqfm"]
Jul 01 17:30:38.426918 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.426900    2576 scope.go:117] "RemoveContainer" containerID="6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777"
Jul 01 17:30:38.427213 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:38.427193    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777\": container with ID starting with 6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777 not found: ID does not exist" containerID="6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777"
Jul 01 17:30:38.427279 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.427225    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777"} err="failed to get container status \"6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777\": rpc error: code = NotFound desc = could not find container \"6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777\": container with ID starting with 6a38144caa793a2c748e98c99b4e43c59751251aa8de4604989a37378ac0a777 not found: ID does not exist"
Jul 01 17:30:38.427279 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.427251    2576 scope.go:117] "RemoveContainer" containerID="d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef"
Jul 01 17:30:38.427509 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:38.427491    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef\": container with ID starting with d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef not found: ID does not exist" containerID="d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef"
Jul 01 17:30:38.427630 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.427611    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef"} err="failed to get container status \"d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef\": rpc error: code = NotFound desc = could not find container \"d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef\": container with ID starting with d564c3831673b582b4bfc746554796614f5c8e4ca297f3090296b808ae5e20ef not found: ID does not exist"
Jul 01 17:30:38.427673 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.427632    2576 scope.go:117] "RemoveContainer" containerID="8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f"
Jul 01 17:30:38.427861 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:30:38.427840    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f\": container with ID starting with 8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f not found: ID does not exist" containerID="8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f"
Jul 01 17:30:38.427922 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:38.427870    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f"} err="failed to get container status \"8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f\": rpc error: code = NotFound desc = could not find container \"8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f\": container with ID starting with 8464cdcbdcee68258cc0fb28e3a6664371e620f59ad412797ffe787b5cc4a20f not found: ID does not exist"
Jul 01 17:30:39.410685 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:39.410636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerStarted","Data":"5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9"}
Jul 01 17:30:39.411088 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:39.410739    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerStarted","Data":"81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca"}
Jul 01 17:30:39.411088 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:39.410942    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:39.432243 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:39.432182    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podStartSLOduration=2.817837851 podStartE2EDuration="6.432161709s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.614323858s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:30:39.429854178 +0000 UTC m=+869.835514956" watchObservedRunningTime="2026-07-01 17:30:39.432161709 +0000 UTC m=+869.837822486"
Jul 01 17:30:40.048176 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:40.048144    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" path="/var/lib/kubelet/pods/21d1992b-eeb2-4852-a0b9-6f6587de84ad/volumes"
Jul 01 17:30:40.415494 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:40.415407    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:40.416757 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:40.416719    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:30:41.420383 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:41.420346    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:30:46.426275 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:46.426245    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:30:46.426788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:46.426757    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:30:56.427200 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:30:56.427156    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:31:06.426859 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:06.426761    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:31:10.017732 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:10.017700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:31:10.018153 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:10.018045    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:31:16.427153 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:16.427110    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:31:26.427219 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:26.427180    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:31:36.427471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:36.427431    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:31:46.427131 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:46.427088    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:31:56.428142 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:31:56.428114    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:32:23.524644 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524609    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"]
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524902    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524913    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524920    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kube-rbac-proxy"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524926    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kube-rbac-proxy"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524938    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kube-rbac-proxy"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524944    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kube-rbac-proxy"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524956    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="storage-initializer"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524961    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="storage-initializer"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524967    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524972    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524978    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="storage-initializer"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.524983    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="storage-initializer"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.525028    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kserve-container"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.525036    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kserve-container"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.525043    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d1992b-eeb2-4852-a0b9-6f6587de84ad" containerName="kube-rbac-proxy"
Jul 01 17:32:23.525148 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.525049    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8581e75-339c-41ac-9673-19613153be80" containerName="kube-rbac-proxy"
Jul 01 17:32:23.526926 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.526907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.529709 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.529684    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-3f4cf-predictor-serving-cert\""
Jul 01 17:32:23.529797 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.529719    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-3f4cf-kube-rbac-proxy-sar-config\""
Jul 01 17:32:23.540191 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.540160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"]
Jul 01 17:32:23.587072 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.587040    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"]
Jul 01 17:32:23.587356 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.587330    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" containerID="cri-o://81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca" gracePeriod=30
Jul 01 17:32:23.587412 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.587357    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kube-rbac-proxy" containerID="cri-o://5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9" gracePeriod=30
Jul 01 17:32:23.619046 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.619019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-proxy-tls\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.619196 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.619070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cngs5\" (UniqueName: \"kubernetes.io/projected/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-api-access-cngs5\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.619263 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.619185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.720631 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.720601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-proxy-tls\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.720761 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.720660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cngs5\" (UniqueName: \"kubernetes.io/projected/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-api-access-cngs5\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.720761 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.720700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.721303 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.721281    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.722981 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.722959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-proxy-tls\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.729538 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.729496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cngs5\" (UniqueName: \"kubernetes.io/projected/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-api-access-cngs5\") pod \"message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") " pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.808540 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.808441    2576 generic.go:356] "Generic (PLEG): container finished" podID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerID="5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9" exitCode=2
Jul 01 17:32:23.808540 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.808501    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerDied","Data":"5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9"}
Jul 01 17:32:23.836788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.836754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:23.963815 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.963791    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"]
Jul 01 17:32:23.966373 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:32:23.966346    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db99ae0_0cee_4c91_8cc7_0d8a6885fba1.slice/crio-3b1abafbc5795076150af8d268a99f2e0925bae23fb49b93d107f02598721353 WatchSource:0}: Error finding container 3b1abafbc5795076150af8d268a99f2e0925bae23fb49b93d107f02598721353: Status 404 returned error can't find the container with id 3b1abafbc5795076150af8d268a99f2e0925bae23fb49b93d107f02598721353
Jul 01 17:32:23.968151 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:23.968134    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 01 17:32:24.813964 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:24.813925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" event={"ID":"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1","Type":"ContainerStarted","Data":"3b1abafbc5795076150af8d268a99f2e0925bae23fb49b93d107f02598721353"}
Jul 01 17:32:25.819438 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:25.819412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" event={"ID":"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1","Type":"ContainerStarted","Data":"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"}
Jul 01 17:32:26.420815 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:26.420773    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 01 17:32:26.426725 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:26.426689    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 01 17:32:26.824900 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:26.824855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" event={"ID":"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1","Type":"ContainerStarted","Data":"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"}
Jul 01 17:32:26.825364 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:26.825001    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:26.844612 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:26.844563    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" podStartSLOduration=2.120139817 podStartE2EDuration="3.844547948s" totalImagesPullingTime="1.724408131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:32:26.843679354 +0000 UTC m=+977.249340131" watchObservedRunningTime="2026-07-01 17:32:26.844547948 +0000 UTC m=+977.250208723"
Jul 01 17:32:27.830539 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:27.830487    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:27.832465 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:27.832440    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:28.434975 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.434943    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:32:28.557129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-rbac-proxy-sar-config\") pod \"369fd6fa-217d-448b-94a2-bf0938246ab5\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") "
Jul 01 17:32:28.557129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557077    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls\") pod \"369fd6fa-217d-448b-94a2-bf0938246ab5\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") "
Jul 01 17:32:28.557129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557101    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/369fd6fa-217d-448b-94a2-bf0938246ab5-kserve-provision-location\") pod \"369fd6fa-217d-448b-94a2-bf0938246ab5\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") "
Jul 01 17:32:28.557129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557126    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l68bk\" (UniqueName: \"kubernetes.io/projected/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-api-access-l68bk\") pod \"369fd6fa-217d-448b-94a2-bf0938246ab5\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") "
Jul 01 17:32:28.557466 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557153    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-cabundle-cert\") pod \"369fd6fa-217d-448b-94a2-bf0938246ab5\" (UID: \"369fd6fa-217d-448b-94a2-bf0938246ab5\") "
Jul 01 17:32:28.557552 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557455    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/369fd6fa-217d-448b-94a2-bf0938246ab5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "369fd6fa-217d-448b-94a2-bf0938246ab5" (UID: "369fd6fa-217d-448b-94a2-bf0938246ab5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:32:28.557552 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557482    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "369fd6fa-217d-448b-94a2-bf0938246ab5" (UID: "369fd6fa-217d-448b-94a2-bf0938246ab5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:32:28.557628 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.557564    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "369fd6fa-217d-448b-94a2-bf0938246ab5" (UID: "369fd6fa-217d-448b-94a2-bf0938246ab5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:32:28.559224 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.559198    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "369fd6fa-217d-448b-94a2-bf0938246ab5" (UID: "369fd6fa-217d-448b-94a2-bf0938246ab5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:32:28.559326 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.559223    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-api-access-l68bk" (OuterVolumeSpecName: "kube-api-access-l68bk") pod "369fd6fa-217d-448b-94a2-bf0938246ab5" (UID: "369fd6fa-217d-448b-94a2-bf0938246ab5"). InnerVolumeSpecName "kube-api-access-l68bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:32:28.658672 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.658625    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/369fd6fa-217d-448b-94a2-bf0938246ab5-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:32:28.658672 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.658666    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/369fd6fa-217d-448b-94a2-bf0938246ab5-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:32:28.658672 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.658679    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68bk\" (UniqueName: \"kubernetes.io/projected/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-api-access-l68bk\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:32:28.658902 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.658691    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:32:28.658902 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.658703    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/369fd6fa-217d-448b-94a2-bf0938246ab5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:32:28.839691 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.839607    2576 generic.go:356] "Generic (PLEG): container finished" podID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerID="81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca" exitCode=0
Jul 01 17:32:28.839691 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.839666    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"
Jul 01 17:32:28.840177 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.839694    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerDied","Data":"81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca"}
Jul 01 17:32:28.840177 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.839761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc" event={"ID":"369fd6fa-217d-448b-94a2-bf0938246ab5","Type":"ContainerDied","Data":"08ddaae7f67df085b1da3f0f3f62b219c5aa75f359e988bfae82a2aba0bac5ee"}
Jul 01 17:32:28.840177 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.839803    2576 scope.go:117] "RemoveContainer" containerID="5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9"
Jul 01 17:32:28.847749 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.847540    2576 scope.go:117] "RemoveContainer" containerID="81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca"
Jul 01 17:32:28.855019 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.855001    2576 scope.go:117] "RemoveContainer" containerID="9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0"
Jul 01 17:32:28.862121 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.862101    2576 scope.go:117] "RemoveContainer" containerID="5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9"
Jul 01 17:32:28.862388 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:32:28.862365    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9\": container with ID starting with 5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9 not found: ID does not exist" containerID="5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9"
Jul 01 17:32:28.862531 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.862395    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9"} err="failed to get container status \"5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9\": rpc error: code = NotFound desc = could not find container \"5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9\": container with ID starting with 5f49c35793a9f2f17761ae76482b76027f6d3ae166612c692b21b6f36cd25da9 not found: ID does not exist"
Jul 01 17:32:28.862531 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.862413    2576 scope.go:117] "RemoveContainer" containerID="81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca"
Jul 01 17:32:28.862931 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:32:28.862699    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca\": container with ID starting with 81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca not found: ID does not exist" containerID="81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca"
Jul 01 17:32:28.862931 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.862729    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca"} err="failed to get container status \"81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca\": rpc error: code = NotFound desc = could not find container \"81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca\": container with ID starting with 81d0f4939a252d08cfa6e4d795eb0d9b2a89a488762af8d86087778304e430ca not found: ID does not exist"
Jul 01 17:32:28.862931 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.862750    2576 scope.go:117] "RemoveContainer" containerID="9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0"
Jul 01 17:32:28.863093 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.862926    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"]
Jul 01 17:32:28.863298 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:32:28.863274    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0\": container with ID starting with 9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0 not found: ID does not exist" containerID="9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0"
Jul 01 17:32:28.863437 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.863406    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0"} err="failed to get container status \"9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0\": rpc error: code = NotFound desc = could not find container \"9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0\": container with ID starting with 9f19db02d7d54bc13142ca2f605ec39b46c24629a3b0599909631555a394cfd0 not found: ID does not exist"
Jul 01 17:32:28.864945 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:28.864927    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-bca43-predictor-85bb49d8cb-8jjlc"]
Jul 01 17:32:30.048236 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:30.048204    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" path="/var/lib/kubelet/pods/369fd6fa-217d-448b-94a2-bf0938246ab5/volumes"
Jul 01 17:32:34.849163 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:34.849082    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:32:43.554105 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554066    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"]
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554361    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container"
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554372    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container"
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554382    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="storage-initializer"
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554387    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="storage-initializer"
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554396    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kube-rbac-proxy"
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554402    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kube-rbac-proxy"
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554451    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kube-rbac-proxy"
Jul 01 17:32:43.554478 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.554459    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="369fd6fa-217d-448b-94a2-bf0938246ab5" containerName="kserve-container"
Jul 01 17:32:43.557586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.557567    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.560404 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.560371    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-3f4cf-predictor-serving-cert\""
Jul 01 17:32:43.560404 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.560394    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:32:43.560665 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.560643    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:32:43.560788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.560680    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-3f4cf-kube-rbac-proxy-sar-config\""
Jul 01 17:32:43.567668 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.567648    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"]
Jul 01 17:32:43.680909 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.680876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlgt\" (UniqueName: \"kubernetes.io/projected/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-api-access-qjlgt\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.681072 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.680925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30af7e8-f085-430d-820a-b00093aa4cf1-proxy-tls\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.681072 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.680977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.681072 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.681027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30af7e8-f085-430d-820a-b00093aa4cf1-kserve-provision-location\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.681072 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.681053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-cabundle-cert\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.781403 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.781367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30af7e8-f085-430d-820a-b00093aa4cf1-proxy-tls\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.781403 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.781404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.781699 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.781441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30af7e8-f085-430d-820a-b00093aa4cf1-kserve-provision-location\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.781699 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.781465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-cabundle-cert\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.781699 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.781487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlgt\" (UniqueName: \"kubernetes.io/projected/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-api-access-qjlgt\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.781978 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.781952    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30af7e8-f085-430d-820a-b00093aa4cf1-kserve-provision-location\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.782171 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.782150    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-cabundle-cert\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.782238 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.782194    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.783921 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.783900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30af7e8-f085-430d-820a-b00093aa4cf1-proxy-tls\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.790254 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.790229    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlgt\" (UniqueName: \"kubernetes.io/projected/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-api-access-qjlgt\") pod \"isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") " pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.868008 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.867926    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:43.997865 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:43.997837    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"]
Jul 01 17:32:44.000067 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:32:44.000035    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30af7e8_f085_430d_820a_b00093aa4cf1.slice/crio-285af7401cf04f761d2e8317d4e9e81bce1d2d8e11c1ad1d7fac9a1463143b7d WatchSource:0}: Error finding container 285af7401cf04f761d2e8317d4e9e81bce1d2d8e11c1ad1d7fac9a1463143b7d: Status 404 returned error can't find the container with id 285af7401cf04f761d2e8317d4e9e81bce1d2d8e11c1ad1d7fac9a1463143b7d
Jul 01 17:32:44.905484 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:44.905444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerStarted","Data":"f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d"}
Jul 01 17:32:44.905892 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:44.905558    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerStarted","Data":"285af7401cf04f761d2e8317d4e9e81bce1d2d8e11c1ad1d7fac9a1463143b7d"}
Jul 01 17:32:48.922146 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:48.922111    2576 generic.go:356] "Generic (PLEG): container finished" podID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerID="f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d" exitCode=0
Jul 01 17:32:48.922538 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:48.922188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerDied","Data":"f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d"}
Jul 01 17:32:49.929233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:49.929196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerStarted","Data":"ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685"}
Jul 01 17:32:49.929617 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:49.929267    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerStarted","Data":"fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488"}
Jul 01 17:32:49.929617 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:49.929292    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerStarted","Data":"c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127"}
Jul 01 17:32:49.929617 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:49.929591    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:49.929752 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:49.929733    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:49.931294 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:49.931268    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:32:49.953540 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:49.953442    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podStartSLOduration=3.076021252 podStartE2EDuration="6.953427697s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877406445s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:32:49.951266942 +0000 UTC m=+1000.356927716" watchObservedRunningTime="2026-07-01 17:32:49.953427697 +0000 UTC m=+1000.359088544"
Jul 01 17:32:50.934541 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:50.934485    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:50.934997 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:50.934679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:32:50.935795 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:50.935768    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:32:51.939626 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:51.939582    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:32:51.940085 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:51.940023    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:32:56.944130 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:56.944097    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:32:56.944757 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:56.944706    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:32:56.944994 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:32:56.944966    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:33:06.945541 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:06.945480    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:33:06.945996 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:06.945969    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:33:16.945249 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:16.945201    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:33:16.945704 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:16.945679    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:33:26.944905 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:26.944857    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:33:26.945336 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:26.945249    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:33:36.944671 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:36.944630    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:33:36.945189 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:36.945163    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:33:46.944681 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:46.944636    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:33:46.945097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:46.945074    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:33:56.944634 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:56.944586    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:33:56.945122 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:33:56.945068    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:34:05.044038 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:05.043957    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:34:05.044468 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:05.044051    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:34:18.590279 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.590232    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw_2db99ae0-0cee-4c91-8cc7-0d8a6885fba1/kserve-container/0.log"
Jul 01 17:34:18.894396 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.894309    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"]
Jul 01 17:34:18.894623 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.894584    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" containerID="cri-o://c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127" gracePeriod=30
Jul 01 17:34:18.894768 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.894613    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" containerID="cri-o://ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685" gracePeriod=30
Jul 01 17:34:18.894768 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.894645    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy" containerID="cri-o://fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488" gracePeriod=30
Jul 01 17:34:18.898325 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.898303    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"]
Jul 01 17:34:18.901833 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.901810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:18.905017 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.905000    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-7e747-predictor-serving-cert\""
Jul 01 17:34:18.905303 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.905285    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-7e747-kube-rbac-proxy-sar-config\""
Jul 01 17:34:18.914874 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.914851    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"]
Jul 01 17:34:18.917873 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.917853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:18.917974 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.917881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:18.917974 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.917907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe3f2a7-414d-4d79-aa60-665639825b0f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:18.917974 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.917960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:18.918091 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.918014    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"]
Jul 01 17:34:18.918091 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.918055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lv7j\" (UniqueName: \"kubernetes.io/projected/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-api-access-7lv7j\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:18.918242 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.918203    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kserve-container" containerID="cri-o://4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5" gracePeriod=30
Jul 01 17:34:18.918302 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:18.918271    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kube-rbac-proxy" containerID="cri-o://3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546" gracePeriod=30
Jul 01 17:34:19.018739 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.018699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lv7j\" (UniqueName: \"kubernetes.io/projected/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-api-access-7lv7j\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.018921 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.018748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.018921 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.018766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.018921 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.018794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe3f2a7-414d-4d79-aa60-665639825b0f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.018921 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.018817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.019125 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:19.018936    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-serving-cert: secret "isvc-sklearn-scale-raw-7e747-predictor-serving-cert" not found
Jul 01 17:34:19.019125 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:19.019018    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls podName:5fe3f2a7-414d-4d79-aa60-665639825b0f nodeName:}" failed. No retries permitted until 2026-07-01 17:34:19.51899525 +0000 UTC m=+1089.924656025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls") pod "isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" (UID: "5fe3f2a7-414d-4d79-aa60-665639825b0f") : secret "isvc-sklearn-scale-raw-7e747-predictor-serving-cert" not found
Jul 01 17:34:19.019233 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.019213    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe3f2a7-414d-4d79-aa60-665639825b0f-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.019429 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.019411    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-cabundle-cert\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.019611 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.019595    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.027229 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.027208    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lv7j\" (UniqueName: \"kubernetes.io/projected/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-api-access-7lv7j\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.160410 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.160387    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:34:19.220931 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.220898    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-proxy-tls\") pod \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") "
Jul 01 17:34:19.221089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.220955    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cngs5\" (UniqueName: \"kubernetes.io/projected/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-api-access-cngs5\") pod \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") "
Jul 01 17:34:19.221089 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.220996    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-rbac-proxy-sar-config\") pod \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\" (UID: \"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1\") "
Jul 01 17:34:19.221433 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.221409    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" (UID: "2db99ae0-0cee-4c91-8cc7-0d8a6885fba1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:34:19.223017 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.222985    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" (UID: "2db99ae0-0cee-4c91-8cc7-0d8a6885fba1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:34:19.223109 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.223014    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-api-access-cngs5" (OuterVolumeSpecName: "kube-api-access-cngs5") pod "2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" (UID: "2db99ae0-0cee-4c91-8cc7-0d8a6885fba1"). InnerVolumeSpecName "kube-api-access-cngs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:34:19.294772 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.294741    2576 generic.go:356] "Generic (PLEG): container finished" podID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerID="fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488" exitCode=2
Jul 01 17:34:19.294942 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.294807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerDied","Data":"fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488"}
Jul 01 17:34:19.296274 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.296253    2576 generic.go:356] "Generic (PLEG): container finished" podID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerID="3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546" exitCode=2
Jul 01 17:34:19.296274 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.296270    2576 generic.go:356] "Generic (PLEG): container finished" podID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerID="4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5" exitCode=2
Jul 01 17:34:19.296415 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.296315    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"
Jul 01 17:34:19.296415 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.296317    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" event={"ID":"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1","Type":"ContainerDied","Data":"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"}
Jul 01 17:34:19.296476 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.296424    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" event={"ID":"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1","Type":"ContainerDied","Data":"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"}
Jul 01 17:34:19.296476 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.296445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw" event={"ID":"2db99ae0-0cee-4c91-8cc7-0d8a6885fba1","Type":"ContainerDied","Data":"3b1abafbc5795076150af8d268a99f2e0925bae23fb49b93d107f02598721353"}
Jul 01 17:34:19.296476 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.296469    2576 scope.go:117] "RemoveContainer" containerID="3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"
Jul 01 17:34:19.304929 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.304911    2576 scope.go:117] "RemoveContainer" containerID="4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"
Jul 01 17:34:19.312240 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.312219    2576 scope.go:117] "RemoveContainer" containerID="3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"
Jul 01 17:34:19.312483 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:19.312465    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546\": container with ID starting with 3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546 not found: ID does not exist" containerID="3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"
Jul 01 17:34:19.312579 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.312495    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"} err="failed to get container status \"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546\": rpc error: code = NotFound desc = could not find container \"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546\": container with ID starting with 3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546 not found: ID does not exist"
Jul 01 17:34:19.312579 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.312533    2576 scope.go:117] "RemoveContainer" containerID="4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"
Jul 01 17:34:19.312780 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:19.312761    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5\": container with ID starting with 4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5 not found: ID does not exist" containerID="4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"
Jul 01 17:34:19.312863 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.312786    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"} err="failed to get container status \"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5\": rpc error: code = NotFound desc = could not find container \"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5\": container with ID starting with 4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5 not found: ID does not exist"
Jul 01 17:34:19.312863 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.312803    2576 scope.go:117] "RemoveContainer" containerID="3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"
Jul 01 17:34:19.313001 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.312985    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546"} err="failed to get container status \"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546\": rpc error: code = NotFound desc = could not find container \"3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546\": container with ID starting with 3d682cae0486bb79e47049a3bcce595cc1450857ed08df2a09cbb91df1d03546 not found: ID does not exist"
Jul 01 17:34:19.313047 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.313002    2576 scope.go:117] "RemoveContainer" containerID="4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"
Jul 01 17:34:19.313229 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.313211    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5"} err="failed to get container status \"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5\": rpc error: code = NotFound desc = could not find container \"4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5\": container with ID starting with 4b9d241e54a4719b89678175a522e8bd5ec4b19a6c5de444d89047f00e6007b5 not found: ID does not exist"
Jul 01 17:34:19.318766 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.318743    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"]
Jul 01 17:34:19.321738 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.321720    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:19.321814 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.321744    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cngs5\" (UniqueName: \"kubernetes.io/projected/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-api-access-cngs5\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:19.321814 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.321761    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:19.322879 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.322863    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-3f4cf-predictor-76dff767fd-qb7jw"]
Jul 01 17:34:19.523304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.523270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.525694 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.525672    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls\") pod \"isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.815783 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.815685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:19.949823 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:19.949468    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"]
Jul 01 17:34:19.952302 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:34:19.952276    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe3f2a7_414d_4d79_aa60_665639825b0f.slice/crio-a9c0775a72497c23d14ce1a0663ee79186b27b25005ee731384cd2808c3d18d6 WatchSource:0}: Error finding container a9c0775a72497c23d14ce1a0663ee79186b27b25005ee731384cd2808c3d18d6: Status 404 returned error can't find the container with id a9c0775a72497c23d14ce1a0663ee79186b27b25005ee731384cd2808c3d18d6
Jul 01 17:34:20.047895 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:20.047862    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" path="/var/lib/kubelet/pods/2db99ae0-0cee-4c91-8cc7-0d8a6885fba1/volumes"
Jul 01 17:34:20.303366 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:20.303328    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerStarted","Data":"bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d"}
Jul 01 17:34:20.303576 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:20.303399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerStarted","Data":"a9c0775a72497c23d14ce1a0663ee79186b27b25005ee731384cd2808c3d18d6"}
Jul 01 17:34:21.940438 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:21.940392    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:34:24.320861 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:24.320822    2576 generic.go:356] "Generic (PLEG): container finished" podID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerID="c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127" exitCode=0
Jul 01 17:34:24.321275 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:24.320894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerDied","Data":"c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127"}
Jul 01 17:34:24.322273 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:24.322251    2576 generic.go:356] "Generic (PLEG): container finished" podID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerID="bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d" exitCode=0
Jul 01 17:34:24.322381 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:24.322287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerDied","Data":"bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d"}
Jul 01 17:34:25.044130 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.044089    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:34:25.044475 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.044454    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:34:25.329640 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.329554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerStarted","Data":"247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696"}
Jul 01 17:34:25.329640 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.329616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerStarted","Data":"da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237"}
Jul 01 17:34:25.330078 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.329881    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:25.330078 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.329970    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:25.331391 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.331364    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:34:25.349724 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:25.349686    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podStartSLOduration=3.78195584 podStartE2EDuration="7.349672973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.567717133s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:34:25.348541077 +0000 UTC m=+1095.754201852" watchObservedRunningTime="2026-07-01 17:34:25.349672973 +0000 UTC m=+1095.755333749"
Jul 01 17:34:26.333501 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:26.333459    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:34:26.939987 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:26.939940    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:34:31.338129 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:31.338074    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:34:31.338741 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:31.338713    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:34:31.940406 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:31.940361    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:34:31.940592 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:31.940575    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:34:35.043490 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:35.043443    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:34:35.043926 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:35.043839    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:34:36.940708 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:36.940655    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:34:41.338718 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:41.338682    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:34:41.940558 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:41.940498    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:34:45.043688 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:45.043636    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 01 17:34:45.044120 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:45.043796    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:34:45.044120 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:45.043929    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 01 17:34:45.044120 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:45.044022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:34:46.940363 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:46.940316    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 01 17:34:49.050537 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.050497    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:34:49.156359 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.156267    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-cabundle-cert\") pod \"d30af7e8-f085-430d-820a-b00093aa4cf1\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") "
Jul 01 17:34:49.156359 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.156305    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-rbac-proxy-sar-config\") pod \"d30af7e8-f085-430d-820a-b00093aa4cf1\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") "
Jul 01 17:34:49.156359 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.156345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjlgt\" (UniqueName: \"kubernetes.io/projected/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-api-access-qjlgt\") pod \"d30af7e8-f085-430d-820a-b00093aa4cf1\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") "
Jul 01 17:34:49.156657 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.156377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30af7e8-f085-430d-820a-b00093aa4cf1-proxy-tls\") pod \"d30af7e8-f085-430d-820a-b00093aa4cf1\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") "
Jul 01 17:34:49.156657 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.156402    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30af7e8-f085-430d-820a-b00093aa4cf1-kserve-provision-location\") pod \"d30af7e8-f085-430d-820a-b00093aa4cf1\" (UID: \"d30af7e8-f085-430d-820a-b00093aa4cf1\") "
Jul 01 17:34:49.156855 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.156812    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d30af7e8-f085-430d-820a-b00093aa4cf1" (UID: "d30af7e8-f085-430d-820a-b00093aa4cf1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:34:49.156918 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.156856    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d30af7e8-f085-430d-820a-b00093aa4cf1" (UID: "d30af7e8-f085-430d-820a-b00093aa4cf1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:34:49.157094 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.157052    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30af7e8-f085-430d-820a-b00093aa4cf1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d30af7e8-f085-430d-820a-b00093aa4cf1" (UID: "d30af7e8-f085-430d-820a-b00093aa4cf1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:34:49.158631 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.158605    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30af7e8-f085-430d-820a-b00093aa4cf1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d30af7e8-f085-430d-820a-b00093aa4cf1" (UID: "d30af7e8-f085-430d-820a-b00093aa4cf1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:34:49.158769 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.158678    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-api-access-qjlgt" (OuterVolumeSpecName: "kube-api-access-qjlgt") pod "d30af7e8-f085-430d-820a-b00093aa4cf1" (UID: "d30af7e8-f085-430d-820a-b00093aa4cf1"). InnerVolumeSpecName "kube-api-access-qjlgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:34:49.257788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.257750    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:49.257788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.257783    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:49.257788 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.257792    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjlgt\" (UniqueName: \"kubernetes.io/projected/d30af7e8-f085-430d-820a-b00093aa4cf1-kube-api-access-qjlgt\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:49.258025 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.257801    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d30af7e8-f085-430d-820a-b00093aa4cf1-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:49.258025 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.257811    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d30af7e8-f085-430d-820a-b00093aa4cf1-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:34:49.438783 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.438692    2576 generic.go:356] "Generic (PLEG): container finished" podID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerID="ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685" exitCode=0
Jul 01 17:34:49.438783 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.438760    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"
Jul 01 17:34:49.438954 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.438759    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerDied","Data":"ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685"}
Jul 01 17:34:49.438954 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.438912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5" event={"ID":"d30af7e8-f085-430d-820a-b00093aa4cf1","Type":"ContainerDied","Data":"285af7401cf04f761d2e8317d4e9e81bce1d2d8e11c1ad1d7fac9a1463143b7d"}
Jul 01 17:34:49.438954 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.438942    2576 scope.go:117] "RemoveContainer" containerID="ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685"
Jul 01 17:34:49.446978 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.446958    2576 scope.go:117] "RemoveContainer" containerID="fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488"
Jul 01 17:34:49.454043 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.454022    2576 scope.go:117] "RemoveContainer" containerID="c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127"
Jul 01 17:34:49.461250 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.461228    2576 scope.go:117] "RemoveContainer" containerID="f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d"
Jul 01 17:34:49.461323 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.461268    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"]
Jul 01 17:34:49.465886 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.465856    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-3f4cf-predictor-bf7f7d547-j6rj5"]
Jul 01 17:34:49.468842 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.468828    2576 scope.go:117] "RemoveContainer" containerID="ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685"
Jul 01 17:34:49.469090 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:49.469070    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685\": container with ID starting with ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685 not found: ID does not exist" containerID="ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685"
Jul 01 17:34:49.469156 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.469102    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685"} err="failed to get container status \"ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685\": rpc error: code = NotFound desc = could not find container \"ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685\": container with ID starting with ac06a58a533ce708ced45175c6f6e802b75fcd75a0773736e3e7d7182932c685 not found: ID does not exist"
Jul 01 17:34:49.469156 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.469127    2576 scope.go:117] "RemoveContainer" containerID="fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488"
Jul 01 17:34:49.469373 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:49.469354    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488\": container with ID starting with fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488 not found: ID does not exist" containerID="fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488"
Jul 01 17:34:49.469410 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.469379    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488"} err="failed to get container status \"fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488\": rpc error: code = NotFound desc = could not find container \"fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488\": container with ID starting with fcb5018cd4246c4a76444d2f85093b11199d05c43e366a10982348b62a84b488 not found: ID does not exist"
Jul 01 17:34:49.469410 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.469394    2576 scope.go:117] "RemoveContainer" containerID="c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127"
Jul 01 17:34:49.469621 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:49.469604    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127\": container with ID starting with c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127 not found: ID does not exist" containerID="c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127"
Jul 01 17:34:49.469695 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.469628    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127"} err="failed to get container status \"c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127\": rpc error: code = NotFound desc = could not find container \"c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127\": container with ID starting with c1350a00f3130a9f15eefa2fc2c1213699c42cd4e25ae23702bb06f311805127 not found: ID does not exist"
Jul 01 17:34:49.469695 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.469646    2576 scope.go:117] "RemoveContainer" containerID="f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d"
Jul 01 17:34:49.469860 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:34:49.469844    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d\": container with ID starting with f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d not found: ID does not exist" containerID="f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d"
Jul 01 17:34:49.469897 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:49.469863    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d"} err="failed to get container status \"f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d\": rpc error: code = NotFound desc = could not find container \"f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d\": container with ID starting with f5acf60ce551b49eedc8bda3ca2862924a95c6c98e76a3ac665abe65ea564a3d not found: ID does not exist"
Jul 01 17:34:50.048221 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:50.048185    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" path="/var/lib/kubelet/pods/d30af7e8-f085-430d-820a-b00093aa4cf1/volumes"
Jul 01 17:34:51.338916 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:34:51.338872    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:35:01.338961 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:35:01.338922    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:35:11.339016 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:35:11.338969    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:35:21.339000 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:35:21.338956    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:35:31.339592 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:35:31.339552    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:35:41.338779 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:35:41.338687    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:35:51.339590 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:35:51.339541    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:35:52.043183 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:35:52.043138    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:36:02.043409 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:02.043360    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:36:10.039108 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:10.039075    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:36:10.041883 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:10.041862    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:36:12.044096 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:12.044044    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:36:22.043814 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:22.043769    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:36:32.043213 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:32.043161    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:36:42.043132 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:42.043092    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:36:52.043505 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:52.043457    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:36:56.048206 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:56.048178    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:36:59.014674 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:59.014631    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"]
Jul 01 17:36:59.017113 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:59.014982    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" containerID="cri-o://da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237" gracePeriod=30
Jul 01 17:36:59.017113 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:59.015016    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kube-rbac-proxy" containerID="cri-o://247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696" gracePeriod=30
Jul 01 17:36:59.938596 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:59.938558    2576 generic.go:356] "Generic (PLEG): container finished" podID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerID="247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696" exitCode=2
Jul 01 17:36:59.938760 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:36:59.938630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerDied","Data":"247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696"}
Jul 01 17:37:01.334669 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:01.334624    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 01 17:37:04.177403 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177364    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"]
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177684    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kube-rbac-proxy"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177696    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kube-rbac-proxy"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177705    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177711    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177718    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177725    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177734    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="storage-initializer"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177740    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="storage-initializer"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177757    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177762    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177769    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kserve-container"
Jul 01 17:37:04.177796 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177774    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kserve-container"
Jul 01 17:37:04.178165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177821    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kserve-container"
Jul 01 17:37:04.178165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177828    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="agent"
Jul 01 17:37:04.178165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177834    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kube-rbac-proxy"
Jul 01 17:37:04.178165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177840    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d30af7e8-f085-430d-820a-b00093aa4cf1" containerName="kube-rbac-proxy"
Jul 01 17:37:04.178165 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.177846    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db99ae0-0cee-4c91-8cc7-0d8a6885fba1" containerName="kserve-container"
Jul 01 17:37:04.180771 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.180754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.183313 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.183295    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-3249f8-predictor-serving-cert\""
Jul 01 17:37:04.183313 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.183299    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-3249f8-kube-rbac-proxy-sar-config\""
Jul 01 17:37:04.191548 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.191509    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"]
Jul 01 17:37:04.267179 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.267145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.267376 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.267197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gns\" (UniqueName: \"kubernetes.io/projected/787d2292-5840-47c8-956c-17dcb66e9545-kube-api-access-48gns\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.267376 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.267218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-cabundle-cert\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.267376 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.267312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-kube-rbac-proxy-sar-config\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.267376 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.267348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787d2292-5840-47c8-956c-17dcb66e9545-kserve-provision-location\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368005 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.367970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368207 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.368041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gns\" (UniqueName: \"kubernetes.io/projected/787d2292-5840-47c8-956c-17dcb66e9545-kube-api-access-48gns\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368207 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.368071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-cabundle-cert\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368207 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.368100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-kube-rbac-proxy-sar-config\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368207 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.368121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787d2292-5840-47c8-956c-17dcb66e9545-kserve-provision-location\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368207 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:37:04.368133    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-3249f8-predictor-serving-cert: secret "isvc-primary-3249f8-predictor-serving-cert" not found
Jul 01 17:37:04.368456 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:37:04.368224    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls podName:787d2292-5840-47c8-956c-17dcb66e9545 nodeName:}" failed. No retries permitted until 2026-07-01 17:37:04.86820131 +0000 UTC m=+1255.273862064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls") pod "isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" (UID: "787d2292-5840-47c8-956c-17dcb66e9545") : secret "isvc-primary-3249f8-predictor-serving-cert" not found
Jul 01 17:37:04.368674 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.368652    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787d2292-5840-47c8-956c-17dcb66e9545-kserve-provision-location\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368868 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.368850    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-kube-rbac-proxy-sar-config\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.368905 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.368882    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-cabundle-cert\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.377288 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.377262    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gns\" (UniqueName: \"kubernetes.io/projected/787d2292-5840-47c8-956c-17dcb66e9545-kube-api-access-48gns\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.872982 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.872945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:04.875419 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:04.875398    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls\") pod \"isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") " pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:05.090882 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:05.090844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:05.220135 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:05.220080    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"]
Jul 01 17:37:05.222165 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:37:05.222134    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787d2292_5840_47c8_956c_17dcb66e9545.slice/crio-f5339acd2764bf00682c02881fc21e777051f419d21ee2e416e69a6031d86b9f WatchSource:0}: Error finding container f5339acd2764bf00682c02881fc21e777051f419d21ee2e416e69a6031d86b9f: Status 404 returned error can't find the container with id f5339acd2764bf00682c02881fc21e777051f419d21ee2e416e69a6031d86b9f
Jul 01 17:37:05.963187 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:05.963149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerStarted","Data":"be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a"}
Jul 01 17:37:05.963410 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:05.963217    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerStarted","Data":"f5339acd2764bf00682c02881fc21e777051f419d21ee2e416e69a6031d86b9f"}
Jul 01 17:37:06.044334 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:06.044286    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 01 17:37:06.334753 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:06.334661    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 01 17:37:08.859927 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.859900    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:37:08.903449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-rbac-proxy-sar-config\") pod \"5fe3f2a7-414d-4d79-aa60-665639825b0f\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") "
Jul 01 17:37:08.903449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903420    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lv7j\" (UniqueName: \"kubernetes.io/projected/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-api-access-7lv7j\") pod \"5fe3f2a7-414d-4d79-aa60-665639825b0f\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") "
Jul 01 17:37:08.903648 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903466    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls\") pod \"5fe3f2a7-414d-4d79-aa60-665639825b0f\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") "
Jul 01 17:37:08.903682 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903641    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe3f2a7-414d-4d79-aa60-665639825b0f-kserve-provision-location\") pod \"5fe3f2a7-414d-4d79-aa60-665639825b0f\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") "
Jul 01 17:37:08.903726 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903714    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-cabundle-cert\") pod \"5fe3f2a7-414d-4d79-aa60-665639825b0f\" (UID: \"5fe3f2a7-414d-4d79-aa60-665639825b0f\") "
Jul 01 17:37:08.903817 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903781    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5fe3f2a7-414d-4d79-aa60-665639825b0f" (UID: "5fe3f2a7-414d-4d79-aa60-665639825b0f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:37:08.903961 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903947    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:37:08.903961 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.903951    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3f2a7-414d-4d79-aa60-665639825b0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5fe3f2a7-414d-4d79-aa60-665639825b0f" (UID: "5fe3f2a7-414d-4d79-aa60-665639825b0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:37:08.904124 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.904085    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5fe3f2a7-414d-4d79-aa60-665639825b0f" (UID: "5fe3f2a7-414d-4d79-aa60-665639825b0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:37:08.905513 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.905489    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-api-access-7lv7j" (OuterVolumeSpecName: "kube-api-access-7lv7j") pod "5fe3f2a7-414d-4d79-aa60-665639825b0f" (UID: "5fe3f2a7-414d-4d79-aa60-665639825b0f"). InnerVolumeSpecName "kube-api-access-7lv7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:37:08.905616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.905574    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5fe3f2a7-414d-4d79-aa60-665639825b0f" (UID: "5fe3f2a7-414d-4d79-aa60-665639825b0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:37:08.977049 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.977016    2576 generic.go:356] "Generic (PLEG): container finished" podID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerID="da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237" exitCode=0
Jul 01 17:37:08.977201 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.977070    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"
Jul 01 17:37:08.977201 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.977104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerDied","Data":"da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237"}
Jul 01 17:37:08.977201 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.977166    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw" event={"ID":"5fe3f2a7-414d-4d79-aa60-665639825b0f","Type":"ContainerDied","Data":"a9c0775a72497c23d14ce1a0663ee79186b27b25005ee731384cd2808c3d18d6"}
Jul 01 17:37:08.977201 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.977193    2576 scope.go:117] "RemoveContainer" containerID="247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696"
Jul 01 17:37:08.985387 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.985370    2576 scope.go:117] "RemoveContainer" containerID="da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237"
Jul 01 17:37:08.992504 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.992483    2576 scope.go:117] "RemoveContainer" containerID="bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d"
Jul 01 17:37:08.999901 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:08.999875    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"]
Jul 01 17:37:09.000249 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.000233    2576 scope.go:117] "RemoveContainer" containerID="247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696"
Jul 01 17:37:09.000500 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:37:09.000480    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696\": container with ID starting with 247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696 not found: ID does not exist" containerID="247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696"
Jul 01 17:37:09.000603 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.000508    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696"} err="failed to get container status \"247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696\": rpc error: code = NotFound desc = could not find container \"247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696\": container with ID starting with 247054277a8e5b52043c23d6d49e46f3f2c6e033e77a01a471df76283e4e5696 not found: ID does not exist"
Jul 01 17:37:09.000653 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.000607    2576 scope.go:117] "RemoveContainer" containerID="da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237"
Jul 01 17:37:09.000867 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:37:09.000830    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237\": container with ID starting with da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237 not found: ID does not exist" containerID="da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237"
Jul 01 17:37:09.000967 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.000862    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237"} err="failed to get container status \"da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237\": rpc error: code = NotFound desc = could not find container \"da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237\": container with ID starting with da5f130bc21468a8070a916471e333faecfbd3d56e8caa3bece27262d9b04237 not found: ID does not exist"
Jul 01 17:37:09.000967 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.000883    2576 scope.go:117] "RemoveContainer" containerID="bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d"
Jul 01 17:37:09.001390 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:37:09.001369    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d\": container with ID starting with bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d not found: ID does not exist" containerID="bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d"
Jul 01 17:37:09.001485 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.001461    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d"} err="failed to get container status \"bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d\": rpc error: code = NotFound desc = could not find container \"bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d\": container with ID starting with bb01b3570954533329ec9351d0eeed3b3fac01d3e59d26b19168db37ffa8e77d not found: ID does not exist"
Jul 01 17:37:09.002705 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.002689    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-7e747-predictor-5c658fb4b6-d2zkw"]
Jul 01 17:37:09.004819 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.004803    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5fe3f2a7-414d-4d79-aa60-665639825b0f-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:37:09.004860 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.004823    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lv7j\" (UniqueName: \"kubernetes.io/projected/5fe3f2a7-414d-4d79-aa60-665639825b0f-kube-api-access-7lv7j\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:37:09.004860 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.004833    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5fe3f2a7-414d-4d79-aa60-665639825b0f-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:37:09.004860 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.004844    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5fe3f2a7-414d-4d79-aa60-665639825b0f-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:37:09.984094 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.984054    2576 generic.go:356] "Generic (PLEG): container finished" podID="787d2292-5840-47c8-956c-17dcb66e9545" containerID="be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a" exitCode=0
Jul 01 17:37:09.984504 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:09.984123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerDied","Data":"be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a"}
Jul 01 17:37:10.048854 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:10.048830    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" path="/var/lib/kubelet/pods/5fe3f2a7-414d-4d79-aa60-665639825b0f/volumes"
Jul 01 17:37:10.991135 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:10.991098    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerStarted","Data":"664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f"}
Jul 01 17:37:10.991555 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:10.991159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerStarted","Data":"cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803"}
Jul 01 17:37:10.991555 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:10.991427    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:10.991555 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:10.991546    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:10.993023 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:10.992994    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:37:11.012074 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:11.012026    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podStartSLOduration=3.042287014 podStartE2EDuration="7.012009795s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.969722781s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:37:11.009559874 +0000 UTC m=+1261.415220651" watchObservedRunningTime="2026-07-01 17:37:11.012009795 +0000 UTC m=+1261.417670570"
Jul 01 17:37:11.995385 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:11.995344    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:37:17.000010 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:16.999979    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:37:17.000461 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:17.000431    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:37:27.001206 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:27.001166    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:37:37.000425 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:37.000387    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:37:47.001222 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:47.001183    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:37:57.001288 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:37:57.001248    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:38:07.001254 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:07.001208    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:38:17.000713 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:17.000669    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:38:25.044092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:25.044060    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:38:34.363923 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.363842    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"]
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364144    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kube-rbac-proxy"
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364154    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kube-rbac-proxy"
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364172    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container"
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364178    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container"
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364186    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="storage-initializer"
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364193    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="storage-initializer"
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364237    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kube-rbac-proxy"
Jul 01 17:38:34.364570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.364247    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe3f2a7-414d-4d79-aa60-665639825b0f" containerName="kserve-container"
Jul 01 17:38:34.367258 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.367242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.369970 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.369948    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-3249f8\""
Jul 01 17:38:34.370092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.369988    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-3249f8-predictor-serving-cert\""
Jul 01 17:38:34.370092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.370013    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-3249f8-kube-rbac-proxy-sar-config\""
Jul 01 17:38:34.370092 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.369991    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-3249f8-dockercfg-925kg\""
Jul 01 17:38:34.377824 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.377803    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"]
Jul 01 17:38:34.453957 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.453921    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.453957 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.453966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-cabundle-cert\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.454168 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.454008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe5ee809-e61b-40d2-b9e1-e39b41427781-proxy-tls\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.454168 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.454027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5ee809-e61b-40d2-b9e1-e39b41427781-kserve-provision-location\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.454168 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.454060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzgc\" (UniqueName: \"kubernetes.io/projected/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-api-access-5wzgc\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555202 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555202 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-cabundle-cert\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe5ee809-e61b-40d2-b9e1-e39b41427781-proxy-tls\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5ee809-e61b-40d2-b9e1-e39b41427781-kserve-provision-location\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555449 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzgc\" (UniqueName: \"kubernetes.io/projected/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-api-access-5wzgc\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555795 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5ee809-e61b-40d2-b9e1-e39b41427781-kserve-provision-location\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555936 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555916    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-cabundle-cert\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.555984 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.555951    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.557861 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.557841    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe5ee809-e61b-40d2-b9e1-e39b41427781-proxy-tls\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.565085 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.565061    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzgc\" (UniqueName: \"kubernetes.io/projected/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-api-access-5wzgc\") pod \"isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") " pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.678584 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.678463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:34.809737 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:34.809712    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"]
Jul 01 17:38:34.811947 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:38:34.811923    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5ee809_e61b_40d2_b9e1_e39b41427781.slice/crio-5b6e1197db7831a8dc3252d34c999d5b8ac0d3f62d90c6a67e5650f656f560e9 WatchSource:0}: Error finding container 5b6e1197db7831a8dc3252d34c999d5b8ac0d3f62d90c6a67e5650f656f560e9: Status 404 returned error can't find the container with id 5b6e1197db7831a8dc3252d34c999d5b8ac0d3f62d90c6a67e5650f656f560e9
Jul 01 17:38:35.307953 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:35.307911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl" event={"ID":"fe5ee809-e61b-40d2-b9e1-e39b41427781","Type":"ContainerStarted","Data":"cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543"}
Jul 01 17:38:35.308119 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:35.307971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl" event={"ID":"fe5ee809-e61b-40d2-b9e1-e39b41427781","Type":"ContainerStarted","Data":"5b6e1197db7831a8dc3252d34c999d5b8ac0d3f62d90c6a67e5650f656f560e9"}
Jul 01 17:38:41.336096 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:41.336065    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_fe5ee809-e61b-40d2-b9e1-e39b41427781/storage-initializer/0.log"
Jul 01 17:38:41.336564 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:41.336100    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerID="cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543" exitCode=1
Jul 01 17:38:41.336564 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:41.336177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl" event={"ID":"fe5ee809-e61b-40d2-b9e1-e39b41427781","Type":"ContainerDied","Data":"cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543"}
Jul 01 17:38:42.341558 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:42.341513    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_fe5ee809-e61b-40d2-b9e1-e39b41427781/storage-initializer/0.log"
Jul 01 17:38:42.341965 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:42.341599    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl" event={"ID":"fe5ee809-e61b-40d2-b9e1-e39b41427781","Type":"ContainerStarted","Data":"550a1d48ef372a2d237bd7d7102b3d0e779bd7c15fea78cad008e5eac56dd357"}
Jul 01 17:38:44.352843 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:44.352814    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_fe5ee809-e61b-40d2-b9e1-e39b41427781/storage-initializer/1.log"
Jul 01 17:38:44.353213 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:44.353185    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_fe5ee809-e61b-40d2-b9e1-e39b41427781/storage-initializer/0.log"
Jul 01 17:38:44.353254 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:44.353217    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerID="550a1d48ef372a2d237bd7d7102b3d0e779bd7c15fea78cad008e5eac56dd357" exitCode=1
Jul 01 17:38:44.353291 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:44.353274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl" event={"ID":"fe5ee809-e61b-40d2-b9e1-e39b41427781","Type":"ContainerDied","Data":"550a1d48ef372a2d237bd7d7102b3d0e779bd7c15fea78cad008e5eac56dd357"}
Jul 01 17:38:44.353351 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:44.353330    2576 scope.go:117] "RemoveContainer" containerID="cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543"
Jul 01 17:38:44.353759 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:44.353740    2576 scope.go:117] "RemoveContainer" containerID="cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543"
Jul 01 17:38:44.363802 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:38:44.363780    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_kserve-ci-e2e-test(fe5ee809-e61b-40d2-b9e1-e39b41427781)\"" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781"
Jul 01 17:38:44.363903 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:38:44.363879    2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_kserve-ci-e2e-test_fe5ee809-e61b-40d2-b9e1-e39b41427781_0 in pod sandbox 5b6e1197db7831a8dc3252d34c999d5b8ac0d3f62d90c6a67e5650f656f560e9 from index: no such id: 'cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543'" containerID="cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543"
Jul 01 17:38:44.363955 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:44.363915    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_kserve-ci-e2e-test_fe5ee809-e61b-40d2-b9e1-e39b41427781_0 in pod sandbox 5b6e1197db7831a8dc3252d34c999d5b8ac0d3f62d90c6a67e5650f656f560e9 from index: no such id: 'cb245505aae257f6fc8bf17ae2778a1ab3412b38f93de52e03b4b876533d9543'"
Jul 01 17:38:45.359284 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:45.359255    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_fe5ee809-e61b-40d2-b9e1-e39b41427781/storage-initializer/1.log"
Jul 01 17:38:52.458265 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.458230    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"]
Jul 01 17:38:52.519640 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.519609    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"]
Jul 01 17:38:52.519867 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.519841    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" containerID="cri-o://cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803" gracePeriod=30
Jul 01 17:38:52.519947 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.519875    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kube-rbac-proxy" containerID="cri-o://664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f" gracePeriod=30
Jul 01 17:38:52.638068 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.638050    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_fe5ee809-e61b-40d2-b9e1-e39b41427781/storage-initializer/1.log"
Jul 01 17:38:52.640095 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.640072    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:52.700084 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wzgc\" (UniqueName: \"kubernetes.io/projected/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-api-access-5wzgc\") pod \"fe5ee809-e61b-40d2-b9e1-e39b41427781\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") "
Jul 01 17:38:52.700309 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700115    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-cabundle-cert\") pod \"fe5ee809-e61b-40d2-b9e1-e39b41427781\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") "
Jul 01 17:38:52.700309 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700137    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5ee809-e61b-40d2-b9e1-e39b41427781-kserve-provision-location\") pod \"fe5ee809-e61b-40d2-b9e1-e39b41427781\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") "
Jul 01 17:38:52.700309 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700167    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe5ee809-e61b-40d2-b9e1-e39b41427781-proxy-tls\") pod \"fe5ee809-e61b-40d2-b9e1-e39b41427781\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") "
Jul 01 17:38:52.700309 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700193    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-rbac-proxy-sar-config\") pod \"fe5ee809-e61b-40d2-b9e1-e39b41427781\" (UID: \"fe5ee809-e61b-40d2-b9e1-e39b41427781\") "
Jul 01 17:38:52.700540 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700479    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe5ee809-e61b-40d2-b9e1-e39b41427781-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe5ee809-e61b-40d2-b9e1-e39b41427781" (UID: "fe5ee809-e61b-40d2-b9e1-e39b41427781"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:38:52.700597 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700560    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe5ee809-e61b-40d2-b9e1-e39b41427781" (UID: "fe5ee809-e61b-40d2-b9e1-e39b41427781"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:38:52.700671 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.700628    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe5ee809-e61b-40d2-b9e1-e39b41427781" (UID: "fe5ee809-e61b-40d2-b9e1-e39b41427781"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:38:52.702301 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.702273    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5ee809-e61b-40d2-b9e1-e39b41427781-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe5ee809-e61b-40d2-b9e1-e39b41427781" (UID: "fe5ee809-e61b-40d2-b9e1-e39b41427781"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:38:52.702417 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.702316    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-api-access-5wzgc" (OuterVolumeSpecName: "kube-api-access-5wzgc") pod "fe5ee809-e61b-40d2-b9e1-e39b41427781" (UID: "fe5ee809-e61b-40d2-b9e1-e39b41427781"). InnerVolumeSpecName "kube-api-access-5wzgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:38:52.801610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.801498    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:52.801610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.801561    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzgc\" (UniqueName: \"kubernetes.io/projected/fe5ee809-e61b-40d2-b9e1-e39b41427781-kube-api-access-5wzgc\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:52.801610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.801572    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe5ee809-e61b-40d2-b9e1-e39b41427781-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:52.801610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.801580    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe5ee809-e61b-40d2-b9e1-e39b41427781-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:52.801610 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:52.801592    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe5ee809-e61b-40d2-b9e1-e39b41427781-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:53.391593 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.391564    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl_fe5ee809-e61b-40d2-b9e1-e39b41427781/storage-initializer/1.log"
Jul 01 17:38:53.391766 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.391674    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"
Jul 01 17:38:53.391766 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.391679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl" event={"ID":"fe5ee809-e61b-40d2-b9e1-e39b41427781","Type":"ContainerDied","Data":"5b6e1197db7831a8dc3252d34c999d5b8ac0d3f62d90c6a67e5650f656f560e9"}
Jul 01 17:38:53.391862 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.391784    2576 scope.go:117] "RemoveContainer" containerID="550a1d48ef372a2d237bd7d7102b3d0e779bd7c15fea78cad008e5eac56dd357"
Jul 01 17:38:53.394007 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.393979    2576 generic.go:356] "Generic (PLEG): container finished" podID="787d2292-5840-47c8-956c-17dcb66e9545" containerID="664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f" exitCode=2
Jul 01 17:38:53.394097 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.394054    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerDied","Data":"664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f"}
Jul 01 17:38:53.427349 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.427317    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"]
Jul 01 17:38:53.431953 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:53.431923    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-3249f8-predictor-75f6fdbfc4-z6lhl"]
Jul 01 17:38:54.047634 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:54.047595    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781" path="/var/lib/kubelet/pods/fe5ee809-e61b-40d2-b9e1-e39b41427781/volumes"
Jul 01 17:38:55.044246 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:55.044205    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 01 17:38:56.996399 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:56.996359    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 01 17:38:57.163226 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.163197    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:38:57.233090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls\") pod \"787d2292-5840-47c8-956c-17dcb66e9545\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") "
Jul 01 17:38:57.233237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233108    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-kube-rbac-proxy-sar-config\") pod \"787d2292-5840-47c8-956c-17dcb66e9545\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") "
Jul 01 17:38:57.233237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233139    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787d2292-5840-47c8-956c-17dcb66e9545-kserve-provision-location\") pod \"787d2292-5840-47c8-956c-17dcb66e9545\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") "
Jul 01 17:38:57.233237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233156    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-cabundle-cert\") pod \"787d2292-5840-47c8-956c-17dcb66e9545\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") "
Jul 01 17:38:57.233237 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233229    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48gns\" (UniqueName: \"kubernetes.io/projected/787d2292-5840-47c8-956c-17dcb66e9545-kube-api-access-48gns\") pod \"787d2292-5840-47c8-956c-17dcb66e9545\" (UID: \"787d2292-5840-47c8-956c-17dcb66e9545\") "
Jul 01 17:38:57.233502 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233481    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "787d2292-5840-47c8-956c-17dcb66e9545" (UID: "787d2292-5840-47c8-956c-17dcb66e9545"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:38:57.233591 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233506    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787d2292-5840-47c8-956c-17dcb66e9545-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "787d2292-5840-47c8-956c-17dcb66e9545" (UID: "787d2292-5840-47c8-956c-17dcb66e9545"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:38:57.233646 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.233609    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "787d2292-5840-47c8-956c-17dcb66e9545" (UID: "787d2292-5840-47c8-956c-17dcb66e9545"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:38:57.235204 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.235181    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "787d2292-5840-47c8-956c-17dcb66e9545" (UID: "787d2292-5840-47c8-956c-17dcb66e9545"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:38:57.235204 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.235185    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787d2292-5840-47c8-956c-17dcb66e9545-kube-api-access-48gns" (OuterVolumeSpecName: "kube-api-access-48gns") pod "787d2292-5840-47c8-956c-17dcb66e9545" (UID: "787d2292-5840-47c8-956c-17dcb66e9545"). InnerVolumeSpecName "kube-api-access-48gns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:38:57.334809 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.334758    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48gns\" (UniqueName: \"kubernetes.io/projected/787d2292-5840-47c8-956c-17dcb66e9545-kube-api-access-48gns\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:57.334809 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.334804    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/787d2292-5840-47c8-956c-17dcb66e9545-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:57.334809 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.334815    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:57.334809 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.334824    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/787d2292-5840-47c8-956c-17dcb66e9545-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:57.335073 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.334833    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/787d2292-5840-47c8-956c-17dcb66e9545-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:38:57.412414 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.412377    2576 generic.go:356] "Generic (PLEG): container finished" podID="787d2292-5840-47c8-956c-17dcb66e9545" containerID="cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803" exitCode=0
Jul 01 17:38:57.412595 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.412430    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"
Jul 01 17:38:57.412595 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.412460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerDied","Data":"cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803"}
Jul 01 17:38:57.412595 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.412538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q" event={"ID":"787d2292-5840-47c8-956c-17dcb66e9545","Type":"ContainerDied","Data":"f5339acd2764bf00682c02881fc21e777051f419d21ee2e416e69a6031d86b9f"}
Jul 01 17:38:57.412595 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.412568    2576 scope.go:117] "RemoveContainer" containerID="664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f"
Jul 01 17:38:57.421313 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.421290    2576 scope.go:117] "RemoveContainer" containerID="cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803"
Jul 01 17:38:57.428630 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.428607    2576 scope.go:117] "RemoveContainer" containerID="be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a"
Jul 01 17:38:57.438182 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.436475    2576 scope.go:117] "RemoveContainer" containerID="664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f"
Jul 01 17:38:57.438182 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.437502    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"]
Jul 01 17:38:57.438633 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:38:57.438250    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f\": container with ID starting with 664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f not found: ID does not exist" containerID="664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f"
Jul 01 17:38:57.438633 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.438298    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f"} err="failed to get container status \"664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f\": rpc error: code = NotFound desc = could not find container \"664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f\": container with ID starting with 664621e2a96d370c01c05dd587cb34909c3d67e852cd0c35798173d055b3007f not found: ID does not exist"
Jul 01 17:38:57.438633 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.438325    2576 scope.go:117] "RemoveContainer" containerID="cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803"
Jul 01 17:38:57.441787 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:38:57.439907    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803\": container with ID starting with cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803 not found: ID does not exist" containerID="cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803"
Jul 01 17:38:57.441787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.439945    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803"} err="failed to get container status \"cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803\": rpc error: code = NotFound desc = could not find container \"cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803\": container with ID starting with cb20f5bed578ba10539c54e2b8ee14d63aebabc67d1e638fd7b6eeebfbb11803 not found: ID does not exist"
Jul 01 17:38:57.441787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.439971    2576 scope.go:117] "RemoveContainer" containerID="be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a"
Jul 01 17:38:57.441787 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.440309    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-3249f8-predictor-57f65bd6fc-5ps2q"]
Jul 01 17:38:57.442030 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:38:57.442002    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a\": container with ID starting with be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a not found: ID does not exist" containerID="be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a"
Jul 01 17:38:57.442239 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.442039    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a"} err="failed to get container status \"be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a\": rpc error: code = NotFound desc = could not find container \"be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a\": container with ID starting with be3ce3f4a37647723fc046d83d6e4fe8375e6a9c4a100363db483753314ad18a not found: ID does not exist"
Jul 01 17:38:57.630090 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.629997    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"]
Jul 01 17:38:57.630390 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630373    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerName="storage-initializer"
Jul 01 17:38:57.630471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630393    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerName="storage-initializer"
Jul 01 17:38:57.630471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630413    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kube-rbac-proxy"
Jul 01 17:38:57.630471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630421    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kube-rbac-proxy"
Jul 01 17:38:57.630471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630435    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerName="storage-initializer"
Jul 01 17:38:57.630471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630444    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerName="storage-initializer"
Jul 01 17:38:57.630471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630455    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container"
Jul 01 17:38:57.630471 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630463    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container"
Jul 01 17:38:57.630846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630479    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="storage-initializer"
Jul 01 17:38:57.630846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630488    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="storage-initializer"
Jul 01 17:38:57.630846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630570    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kserve-container"
Jul 01 17:38:57.630846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630587    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerName="storage-initializer"
Jul 01 17:38:57.630846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630596    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe5ee809-e61b-40d2-b9e1-e39b41427781" containerName="storage-initializer"
Jul 01 17:38:57.630846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.630605    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="787d2292-5840-47c8-956c-17dcb66e9545" containerName="kube-rbac-proxy"
Jul 01 17:38:57.635126 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.635104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.637859 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.637828    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9c53ba-predictor-serving-cert\""
Jul 01 17:38:57.637984 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.637963    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-9c53ba-dockercfg-ppt7n\""
Jul 01 17:38:57.638277 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.638249    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:38:57.638379 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.638285    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-9c53ba-kube-rbac-proxy-sar-config\""
Jul 01 17:38:57.638379 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.638335    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:38:57.638379 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.638338    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:38:57.638555 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.638338    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-9c53ba\""
Jul 01 17:38:57.643650 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.643626    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"]
Jul 01 17:38:57.740344 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.740302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb719383-a04b-4c16-aa28-80364577ea3e-kserve-provision-location\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.740344 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.740342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb719383-a04b-4c16-aa28-80364577ea3e-proxy-tls\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.740580 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.740380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4j4\" (UniqueName: \"kubernetes.io/projected/eb719383-a04b-4c16-aa28-80364577ea3e-kube-api-access-4d4j4\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.740580 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.740458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-cabundle-cert\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.740580 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.740562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.840935 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.840904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.840935 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.840945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb719383-a04b-4c16-aa28-80364577ea3e-kserve-provision-location\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.841140 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.840971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb719383-a04b-4c16-aa28-80364577ea3e-proxy-tls\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.841140 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.841008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4j4\" (UniqueName: \"kubernetes.io/projected/eb719383-a04b-4c16-aa28-80364577ea3e-kube-api-access-4d4j4\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.841140 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.841043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-cabundle-cert\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.841453 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.841421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb719383-a04b-4c16-aa28-80364577ea3e-kserve-provision-location\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.841769 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.841745    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.841817 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.841801    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-cabundle-cert\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.843465 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.843443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb719383-a04b-4c16-aa28-80364577ea3e-proxy-tls\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.849398 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.849377    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4j4\" (UniqueName: \"kubernetes.io/projected/eb719383-a04b-4c16-aa28-80364577ea3e-kube-api-access-4d4j4\") pod \"isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") " pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:57.946343 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:57.946256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:38:58.049257 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:58.049217    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="787d2292-5840-47c8-956c-17dcb66e9545" path="/var/lib/kubelet/pods/787d2292-5840-47c8-956c-17dcb66e9545/volumes"
Jul 01 17:38:58.079071 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:58.078858    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"]
Jul 01 17:38:58.081366 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:38:58.081343    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb719383_a04b_4c16_aa28_80364577ea3e.slice/crio-5cb1a454622eedc175e18a7608852e4b78d624628b9c06ccf3a1bca7cda910a0 WatchSource:0}: Error finding container 5cb1a454622eedc175e18a7608852e4b78d624628b9c06ccf3a1bca7cda910a0: Status 404 returned error can't find the container with id 5cb1a454622eedc175e18a7608852e4b78d624628b9c06ccf3a1bca7cda910a0
Jul 01 17:38:58.417831 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:58.417789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t" event={"ID":"eb719383-a04b-4c16-aa28-80364577ea3e","Type":"ContainerStarted","Data":"9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598"}
Jul 01 17:38:58.418032 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:38:58.417855    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t" event={"ID":"eb719383-a04b-4c16-aa28-80364577ea3e","Type":"ContainerStarted","Data":"5cb1a454622eedc175e18a7608852e4b78d624628b9c06ccf3a1bca7cda910a0"}
Jul 01 17:39:03.446488 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:03.446457    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_eb719383-a04b-4c16-aa28-80364577ea3e/storage-initializer/0.log"
Jul 01 17:39:03.446891 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:03.446495    2576 generic.go:356] "Generic (PLEG): container finished" podID="eb719383-a04b-4c16-aa28-80364577ea3e" containerID="9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598" exitCode=1
Jul 01 17:39:03.446891 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:03.446580    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t" event={"ID":"eb719383-a04b-4c16-aa28-80364577ea3e","Type":"ContainerDied","Data":"9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598"}
Jul 01 17:39:04.455206 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:04.455180    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_eb719383-a04b-4c16-aa28-80364577ea3e/storage-initializer/0.log"
Jul 01 17:39:04.455643 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:04.455247    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t" event={"ID":"eb719383-a04b-4c16-aa28-80364577ea3e","Type":"ContainerStarted","Data":"65cbe8581fdf13acf27c489be1f776d09798d5a4f58c1056825777c829f806da"}
Jul 01 17:39:06.464648 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:06.464624    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_eb719383-a04b-4c16-aa28-80364577ea3e/storage-initializer/1.log"
Jul 01 17:39:06.465118 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:06.465047    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_eb719383-a04b-4c16-aa28-80364577ea3e/storage-initializer/0.log"
Jul 01 17:39:06.465118 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:06.465094    2576 generic.go:356] "Generic (PLEG): container finished" podID="eb719383-a04b-4c16-aa28-80364577ea3e" containerID="65cbe8581fdf13acf27c489be1f776d09798d5a4f58c1056825777c829f806da" exitCode=1
Jul 01 17:39:06.465232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:06.465141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t" event={"ID":"eb719383-a04b-4c16-aa28-80364577ea3e","Type":"ContainerDied","Data":"65cbe8581fdf13acf27c489be1f776d09798d5a4f58c1056825777c829f806da"}
Jul 01 17:39:06.465232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:06.465212    2576 scope.go:117] "RemoveContainer" containerID="9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598"
Jul 01 17:39:06.465631 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:06.465602    2576 scope.go:117] "RemoveContainer" containerID="9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598"
Jul 01 17:39:06.475511 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:39:06.475486    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_kserve-ci-e2e-test(eb719383-a04b-4c16-aa28-80364577ea3e)\"" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t" podUID="eb719383-a04b-4c16-aa28-80364577ea3e"
Jul 01 17:39:06.475605 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:39:06.475584    2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_kserve-ci-e2e-test_eb719383-a04b-4c16-aa28-80364577ea3e_0 in pod sandbox 5cb1a454622eedc175e18a7608852e4b78d624628b9c06ccf3a1bca7cda910a0 from index: no such id: '9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598'" containerID="9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598"
Jul 01 17:39:06.475651 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:06.475616    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_kserve-ci-e2e-test_eb719383-a04b-4c16-aa28-80364577ea3e_0 in pod sandbox 5cb1a454622eedc175e18a7608852e4b78d624628b9c06ccf3a1bca7cda910a0 from index: no such id: '9f0b00862c7cc3f5683e341e51f5b2b85171dc411d2273877c17c84798b95598'"
Jul 01 17:39:07.470838 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.470808    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_eb719383-a04b-4c16-aa28-80364577ea3e/storage-initializer/1.log"
Jul 01 17:39:07.629873 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.629844    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"]
Jul 01 17:39:07.754672 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.754649    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_eb719383-a04b-4c16-aa28-80364577ea3e/storage-initializer/1.log"
Jul 01 17:39:07.758268 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.758250    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:39:07.921252 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921209    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb719383-a04b-4c16-aa28-80364577ea3e-kserve-provision-location\") pod \"eb719383-a04b-4c16-aa28-80364577ea3e\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") "
Jul 01 17:39:07.921430 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921268    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-cabundle-cert\") pod \"eb719383-a04b-4c16-aa28-80364577ea3e\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") "
Jul 01 17:39:07.921430 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921285    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-kube-rbac-proxy-sar-config\") pod \"eb719383-a04b-4c16-aa28-80364577ea3e\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") "
Jul 01 17:39:07.921430 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921312    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d4j4\" (UniqueName: \"kubernetes.io/projected/eb719383-a04b-4c16-aa28-80364577ea3e-kube-api-access-4d4j4\") pod \"eb719383-a04b-4c16-aa28-80364577ea3e\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") "
Jul 01 17:39:07.921430 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921337    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb719383-a04b-4c16-aa28-80364577ea3e-proxy-tls\") pod \"eb719383-a04b-4c16-aa28-80364577ea3e\" (UID: \"eb719383-a04b-4c16-aa28-80364577ea3e\") "
Jul 01 17:39:07.921664 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921615    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb719383-a04b-4c16-aa28-80364577ea3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb719383-a04b-4c16-aa28-80364577ea3e" (UID: "eb719383-a04b-4c16-aa28-80364577ea3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:39:07.921748 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921730    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb719383-a04b-4c16-aa28-80364577ea3e" (UID: "eb719383-a04b-4c16-aa28-80364577ea3e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:39:07.921806 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.921751    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb719383-a04b-4c16-aa28-80364577ea3e" (UID: "eb719383-a04b-4c16-aa28-80364577ea3e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:39:07.923446 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.923425    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb719383-a04b-4c16-aa28-80364577ea3e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb719383-a04b-4c16-aa28-80364577ea3e" (UID: "eb719383-a04b-4c16-aa28-80364577ea3e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:39:07.923446 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:07.923435    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb719383-a04b-4c16-aa28-80364577ea3e-kube-api-access-4d4j4" (OuterVolumeSpecName: "kube-api-access-4d4j4") pod "eb719383-a04b-4c16-aa28-80364577ea3e" (UID: "eb719383-a04b-4c16-aa28-80364577ea3e"). InnerVolumeSpecName "kube-api-access-4d4j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:39:08.022340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.022248    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb719383-a04b-4c16-aa28-80364577ea3e-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:39:08.022340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.022283    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb719383-a04b-4c16-aa28-80364577ea3e-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:39:08.022340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.022293    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:39:08.022340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.022303    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb719383-a04b-4c16-aa28-80364577ea3e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:39:08.022340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.022313    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4j4\" (UniqueName: \"kubernetes.io/projected/eb719383-a04b-4c16-aa28-80364577ea3e-kube-api-access-4d4j4\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:39:08.476426 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.476399    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t_eb719383-a04b-4c16-aa28-80364577ea3e/storage-initializer/1.log"
Jul 01 17:39:08.476928 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.476493    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"
Jul 01 17:39:08.476928 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.476538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t" event={"ID":"eb719383-a04b-4c16-aa28-80364577ea3e","Type":"ContainerDied","Data":"5cb1a454622eedc175e18a7608852e4b78d624628b9c06ccf3a1bca7cda910a0"}
Jul 01 17:39:08.476928 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.476646    2576 scope.go:117] "RemoveContainer" containerID="65cbe8581fdf13acf27c489be1f776d09798d5a4f58c1056825777c829f806da"
Jul 01 17:39:08.511667 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.511632    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"]
Jul 01 17:39:08.514689 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:08.514662    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-9c53ba-predictor-b5dc898d6-jvs5t"]
Jul 01 17:39:10.048080 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:10.048039    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb719383-a04b-4c16-aa28-80364577ea3e" path="/var/lib/kubelet/pods/eb719383-a04b-4c16-aa28-80364577ea3e/volumes"
Jul 01 17:39:12.806538 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.806494    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"]
Jul 01 17:39:12.806898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.806792    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb719383-a04b-4c16-aa28-80364577ea3e" containerName="storage-initializer"
Jul 01 17:39:12.806898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.806803    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb719383-a04b-4c16-aa28-80364577ea3e" containerName="storage-initializer"
Jul 01 17:39:12.806898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.806817    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb719383-a04b-4c16-aa28-80364577ea3e" containerName="storage-initializer"
Jul 01 17:39:12.806898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.806823    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb719383-a04b-4c16-aa28-80364577ea3e" containerName="storage-initializer"
Jul 01 17:39:12.806898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.806869    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb719383-a04b-4c16-aa28-80364577ea3e" containerName="storage-initializer"
Jul 01 17:39:12.806898 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.806876    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb719383-a04b-4c16-aa28-80364577ea3e" containerName="storage-initializer"
Jul 01 17:39:12.809961 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.809940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:12.812786 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.812745    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:39:12.812786 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.812775    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:39:12.813006 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.812808    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:39:12.813006 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.812897    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:39:12.813110 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.813071    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-26979-kube-rbac-proxy-sar-config\""
Jul 01 17:39:12.814091 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.814072    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-26979-predictor-serving-cert\""
Jul 01 17:39:12.814168 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.814077    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xldfq\""
Jul 01 17:39:12.822617 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.822596    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"]
Jul 01 17:39:12.965774 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.965725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5abc563-238e-48c2-ae3d-2709ce83bdba-kserve-provision-location\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:12.965989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.965790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:12.965989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.965870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:12.965989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.965924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-cabundle-cert\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:12.965989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:12.965980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsns\" (UniqueName: \"kubernetes.io/projected/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-api-access-rdsns\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.067340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.067239    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.067340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.067286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.067340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.067309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-cabundle-cert\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.067340 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.067332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsns\" (UniqueName: \"kubernetes.io/projected/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-api-access-rdsns\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.067759 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.067366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5abc563-238e-48c2-ae3d-2709ce83bdba-kserve-provision-location\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.067759 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:39:13.067416    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-26979-predictor-serving-cert: secret "raw-sklearn-26979-predictor-serving-cert" not found
Jul 01 17:39:13.067759 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:39:13.067507    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls podName:b5abc563-238e-48c2-ae3d-2709ce83bdba nodeName:}" failed. No retries permitted until 2026-07-01 17:39:13.567484403 +0000 UTC m=+1383.973145157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls") pod "raw-sklearn-26979-predictor-7db56d9d88-xbp9m" (UID: "b5abc563-238e-48c2-ae3d-2709ce83bdba") : secret "raw-sklearn-26979-predictor-serving-cert" not found
Jul 01 17:39:13.067958 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.067912    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5abc563-238e-48c2-ae3d-2709ce83bdba-kserve-provision-location\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.068220 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.068192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.068324 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.068268    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-cabundle-cert\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.076128 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.076100    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsns\" (UniqueName: \"kubernetes.io/projected/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-api-access-rdsns\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.571897 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.571857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.574348 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.574322    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls\") pod \"raw-sklearn-26979-predictor-7db56d9d88-xbp9m\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") " pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.719937 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.719894    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:13.858410 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:13.858288    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"]
Jul 01 17:39:13.860603 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:39:13.860572    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5abc563_238e_48c2_ae3d_2709ce83bdba.slice/crio-3919fa2b59d607431b79eafdf767de3e87cc69c61af480b8ac2dbe09bae82ce4 WatchSource:0}: Error finding container 3919fa2b59d607431b79eafdf767de3e87cc69c61af480b8ac2dbe09bae82ce4: Status 404 returned error can't find the container with id 3919fa2b59d607431b79eafdf767de3e87cc69c61af480b8ac2dbe09bae82ce4
Jul 01 17:39:14.500315 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:14.500279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerStarted","Data":"60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b"}
Jul 01 17:39:14.500482 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:14.500337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerStarted","Data":"3919fa2b59d607431b79eafdf767de3e87cc69c61af480b8ac2dbe09bae82ce4"}
Jul 01 17:39:18.516573 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:18.516540    2576 generic.go:356] "Generic (PLEG): container finished" podID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerID="60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b" exitCode=0
Jul 01 17:39:18.516573 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:18.516577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerDied","Data":"60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b"}
Jul 01 17:39:19.522365 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:19.522322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerStarted","Data":"4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5"}
Jul 01 17:39:19.522792 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:19.522416    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerStarted","Data":"e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4"}
Jul 01 17:39:19.522792 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:19.522690    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:19.522792 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:19.522744    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:19.524157 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:19.524131    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:39:19.542231 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:19.542178    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podStartSLOduration=3.748326723 podStartE2EDuration="7.542164257s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793837534s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:39:19.541625421 +0000 UTC m=+1389.947286197" watchObservedRunningTime="2026-07-01 17:39:19.542164257 +0000 UTC m=+1389.947825106"
Jul 01 17:39:20.526547 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:20.526494    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:39:25.531588 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:25.531551    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:39:25.532169 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:25.532142    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:39:35.532752 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:35.532712    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:39:45.532339 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:45.532292    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:39:55.532703 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:39:55.532656    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:40:05.533026 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:05.532983    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:40:15.532350 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:15.532309    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:40:25.532260 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:25.532219    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:40:30.045175 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:30.045135    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 01 17:40:40.047226 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:40.047197    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:40:42.986150 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:42.986117    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"]
Jul 01 17:40:42.986550 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:42.986346    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container" containerID="cri-o://e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4" gracePeriod=30
Jul 01 17:40:42.986550 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:42.986372    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kube-rbac-proxy" containerID="cri-o://4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5" gracePeriod=30
Jul 01 17:40:43.850145 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:43.850103    2576 generic.go:356] "Generic (PLEG): container finished" podID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerID="4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5" exitCode=2
Jul 01 17:40:43.850312 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:43.850153    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerDied","Data":"4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5"}
Jul 01 17:40:45.526885 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:45.526839    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 01 17:40:47.733480 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.733451    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:40:47.810028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.809936    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-rbac-proxy-sar-config\") pod \"b5abc563-238e-48c2-ae3d-2709ce83bdba\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") "
Jul 01 17:40:47.810028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.810004    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdsns\" (UniqueName: \"kubernetes.io/projected/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-api-access-rdsns\") pod \"b5abc563-238e-48c2-ae3d-2709ce83bdba\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") "
Jul 01 17:40:47.810028 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.810027    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-cabundle-cert\") pod \"b5abc563-238e-48c2-ae3d-2709ce83bdba\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") "
Jul 01 17:40:47.810240 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.810090    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5abc563-238e-48c2-ae3d-2709ce83bdba-kserve-provision-location\") pod \"b5abc563-238e-48c2-ae3d-2709ce83bdba\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") "
Jul 01 17:40:47.810240 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.810108    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls\") pod \"b5abc563-238e-48c2-ae3d-2709ce83bdba\" (UID: \"b5abc563-238e-48c2-ae3d-2709ce83bdba\") "
Jul 01 17:40:47.810387 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.810357    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5abc563-238e-48c2-ae3d-2709ce83bdba" (UID: "b5abc563-238e-48c2-ae3d-2709ce83bdba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:40:47.810451 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.810406    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abc563-238e-48c2-ae3d-2709ce83bdba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5abc563-238e-48c2-ae3d-2709ce83bdba" (UID: "b5abc563-238e-48c2-ae3d-2709ce83bdba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:40:47.810451 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.810414    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5abc563-238e-48c2-ae3d-2709ce83bdba" (UID: "b5abc563-238e-48c2-ae3d-2709ce83bdba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:40:47.812176 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.812150    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-api-access-rdsns" (OuterVolumeSpecName: "kube-api-access-rdsns") pod "b5abc563-238e-48c2-ae3d-2709ce83bdba" (UID: "b5abc563-238e-48c2-ae3d-2709ce83bdba"). InnerVolumeSpecName "kube-api-access-rdsns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:40:47.812249 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.812234    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5abc563-238e-48c2-ae3d-2709ce83bdba" (UID: "b5abc563-238e-48c2-ae3d-2709ce83bdba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:40:47.868266 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.868234    2576 generic.go:356] "Generic (PLEG): container finished" podID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerID="e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4" exitCode=0
Jul 01 17:40:47.868453 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.868292    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerDied","Data":"e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4"}
Jul 01 17:40:47.868453 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.868305    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"
Jul 01 17:40:47.868453 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.868341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m" event={"ID":"b5abc563-238e-48c2-ae3d-2709ce83bdba","Type":"ContainerDied","Data":"3919fa2b59d607431b79eafdf767de3e87cc69c61af480b8ac2dbe09bae82ce4"}
Jul 01 17:40:47.868453 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.868367    2576 scope.go:117] "RemoveContainer" containerID="4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5"
Jul 01 17:40:47.877130 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.877115    2576 scope.go:117] "RemoveContainer" containerID="e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4"
Jul 01 17:40:47.884136 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.884120    2576 scope.go:117] "RemoveContainer" containerID="60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b"
Jul 01 17:40:47.891124 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.891101    2576 scope.go:117] "RemoveContainer" containerID="4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5"
Jul 01 17:40:47.891394 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:40:47.891371    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5\": container with ID starting with 4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5 not found: ID does not exist" containerID="4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5"
Jul 01 17:40:47.891474 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.891407    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5"} err="failed to get container status \"4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5\": rpc error: code = NotFound desc = could not find container \"4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5\": container with ID starting with 4be0c9e396d276e94ac5b7bcf6d2cd3ec036e7598926356d8e856900314f59b5 not found: ID does not exist"
Jul 01 17:40:47.891474 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.891433    2576 scope.go:117] "RemoveContainer" containerID="e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4"
Jul 01 17:40:47.891683 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:40:47.891664    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4\": container with ID starting with e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4 not found: ID does not exist" containerID="e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4"
Jul 01 17:40:47.891732 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.891695    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4"} err="failed to get container status \"e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4\": rpc error: code = NotFound desc = could not find container \"e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4\": container with ID starting with e9d839bc2c52f306413afd558bb2f997ba22b9a68a08da3759b8743da52f95d4 not found: ID does not exist"
Jul 01 17:40:47.891732 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.891718    2576 scope.go:117] "RemoveContainer" containerID="60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b"
Jul 01 17:40:47.891937 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:40:47.891919    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b\": container with ID starting with 60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b not found: ID does not exist" containerID="60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b"
Jul 01 17:40:47.891998 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.891944    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b"} err="failed to get container status \"60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b\": rpc error: code = NotFound desc = could not find container \"60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b\": container with ID starting with 60fd01fa935ef1f20fa30341dcd079a6c11f31dc73100d199004506a930d991b not found: ID does not exist"
Jul 01 17:40:47.891998 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.891964    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"]
Jul 01 17:40:47.895951 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.895930    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-26979-predictor-7db56d9d88-xbp9m"]
Jul 01 17:40:47.910888 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.910863    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:40:47.910888 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.910885    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdsns\" (UniqueName: \"kubernetes.io/projected/b5abc563-238e-48c2-ae3d-2709ce83bdba-kube-api-access-rdsns\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:40:47.911008 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.910896    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5abc563-238e-48c2-ae3d-2709ce83bdba-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:40:47.911008 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.910907    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5abc563-238e-48c2-ae3d-2709ce83bdba-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:40:47.911008 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:47.910916    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5abc563-238e-48c2-ae3d-2709ce83bdba-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:40:48.048367 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.048338    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" path="/var/lib/kubelet/pods/b5abc563-238e-48c2-ae3d-2709ce83bdba/volumes"
Jul 01 17:40:48.124621 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124537    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"]
Jul 01 17:40:48.124831 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124818    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container"
Jul 01 17:40:48.124889 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124833    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container"
Jul 01 17:40:48.124889 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124846    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="storage-initializer"
Jul 01 17:40:48.124889 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124852    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="storage-initializer"
Jul 01 17:40:48.124889 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124867    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kube-rbac-proxy"
Jul 01 17:40:48.124889 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kube-rbac-proxy"
Jul 01 17:40:48.125060 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124913    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kserve-container"
Jul 01 17:40:48.125060 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.124920    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5abc563-238e-48c2-ae3d-2709ce83bdba" containerName="kube-rbac-proxy"
Jul 01 17:40:48.129322 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.129299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.133702 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.133681    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-bc58e-kube-rbac-proxy-sar-config\""
Jul 01 17:40:48.133878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.133681    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 01 17:40:48.133878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.133687    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 01 17:40:48.133878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.133837    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 01 17:40:48.134150 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.134133    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 01 17:40:48.134234 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.134165    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-bc58e-predictor-serving-cert\""
Jul 01 17:40:48.134422 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.134407    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xldfq\""
Jul 01 17:40:48.140381 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.140358    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"]
Jul 01 17:40:48.213972 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.213932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e9153bf-6d19-4aca-b797-60a3ce03674c-kserve-provision-location\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.213972 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.213975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.214186 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.213996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f25dn\" (UniqueName: \"kubernetes.io/projected/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-api-access-f25dn\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.214186 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.214068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.214186 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.214104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-cabundle-cert\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315108 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e9153bf-6d19-4aca-b797-60a3ce03674c-kserve-provision-location\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315253 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315253 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f25dn\" (UniqueName: \"kubernetes.io/projected/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-api-access-f25dn\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315253 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315253 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-cabundle-cert\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315447 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:40:48.315414    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-serving-cert: secret "raw-sklearn-runtime-bc58e-predictor-serving-cert" not found
Jul 01 17:40:48.315514 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315493    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e9153bf-6d19-4aca-b797-60a3ce03674c-kserve-provision-location\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315607 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:40:48.315510    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls podName:1e9153bf-6d19-4aca-b797-60a3ce03674c nodeName:}" failed. No retries permitted until 2026-07-01 17:40:48.815486864 +0000 UTC m=+1479.221147632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls") pod "raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" (UID: "1e9153bf-6d19-4aca-b797-60a3ce03674c") : secret "raw-sklearn-runtime-bc58e-predictor-serving-cert" not found
Jul 01 17:40:48.315866 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315848    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.315931 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.315869    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-cabundle-cert\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.326225 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.326205    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f25dn\" (UniqueName: \"kubernetes.io/projected/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-api-access-f25dn\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.819362 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.819303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:48.821712 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:48.821691    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls\") pod \"raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:49.040083 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:49.040046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:49.176998 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:49.176963    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"]
Jul 01 17:40:49.179802 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:40:49.179775    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9153bf_6d19_4aca_b797_60a3ce03674c.slice/crio-72b6ffb99e6b1a9a632428d8f4db452929385eb8c77c9a65dc2fb936fc738aed WatchSource:0}: Error finding container 72b6ffb99e6b1a9a632428d8f4db452929385eb8c77c9a65dc2fb936fc738aed: Status 404 returned error can't find the container with id 72b6ffb99e6b1a9a632428d8f4db452929385eb8c77c9a65dc2fb936fc738aed
Jul 01 17:40:49.879167 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:49.879126    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerStarted","Data":"a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0"}
Jul 01 17:40:49.879592 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:49.879231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerStarted","Data":"72b6ffb99e6b1a9a632428d8f4db452929385eb8c77c9a65dc2fb936fc738aed"}
Jul 01 17:40:52.891738 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:52.891706    2576 generic.go:356] "Generic (PLEG): container finished" podID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerID="a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0" exitCode=0
Jul 01 17:40:52.892088 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:52.891780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerDied","Data":"a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0"}
Jul 01 17:40:53.897729 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:53.897691    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerStarted","Data":"2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818"}
Jul 01 17:40:53.898232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:53.897761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerStarted","Data":"2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5"}
Jul 01 17:40:53.898232 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:53.897974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:53.917563 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:53.917496    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podStartSLOduration=2.364413572 podStartE2EDuration="5.917481892s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.55306832s" isStatefulPod=false podCreationTimestamp="2026-07-01 17:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:40:53.916762816 +0000 UTC m=+1484.322423605" watchObservedRunningTime="2026-07-01 17:40:53.917481892 +0000 UTC m=+1484.323142668"
Jul 01 17:40:54.901682 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:54.901646    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:40:54.903023 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:54.902995    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:40:55.905739 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:40:55.905690    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:41:00.910277 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:00.910250    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:41:00.910783 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:00.910695    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:41:10.063660 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:10.063631    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:41:10.065692 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:10.065671    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:41:10.911139 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:10.911099    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:41:20.911018 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:20.910972    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:41:30.911143 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:30.911102    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:41:40.911579 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:40.911477    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:41:50.911632 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:41:50.911593    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:42:00.911204 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:00.911165    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:42:09.044387 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:09.044357    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:42:18.237206 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:18.237173    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"]
Jul 01 17:42:18.237741 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:18.237441    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" containerID="cri-o://2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5" gracePeriod=30
Jul 01 17:42:18.237741 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:18.237501    2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kube-rbac-proxy" containerID="cri-o://2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818" gracePeriod=30
Jul 01 17:42:19.044131 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:19.044088    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 01 17:42:19.224883 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:19.224843    2576 generic.go:356] "Generic (PLEG): container finished" podID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerID="2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818" exitCode=2
Jul 01 17:42:19.224883 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:19.224885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerDied","Data":"2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818"}
Jul 01 17:42:20.906388 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:20.906343    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 01 17:42:22.969812 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:22.969785    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:42:23.104570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104463    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-cabundle-cert\") pod \"1e9153bf-6d19-4aca-b797-60a3ce03674c\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") "
Jul 01 17:42:23.104570 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104559    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls\") pod \"1e9153bf-6d19-4aca-b797-60a3ce03674c\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") "
Jul 01 17:42:23.104803 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104588    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-rbac-proxy-sar-config\") pod \"1e9153bf-6d19-4aca-b797-60a3ce03674c\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") "
Jul 01 17:42:23.104803 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104618    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e9153bf-6d19-4aca-b797-60a3ce03674c-kserve-provision-location\") pod \"1e9153bf-6d19-4aca-b797-60a3ce03674c\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") "
Jul 01 17:42:23.104803 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104650    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f25dn\" (UniqueName: \"kubernetes.io/projected/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-api-access-f25dn\") pod \"1e9153bf-6d19-4aca-b797-60a3ce03674c\" (UID: \"1e9153bf-6d19-4aca-b797-60a3ce03674c\") "
Jul 01 17:42:23.104932 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104851    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e9153bf-6d19-4aca-b797-60a3ce03674c" (UID: "1e9153bf-6d19-4aca-b797-60a3ce03674c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:42:23.104965 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104933    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9153bf-6d19-4aca-b797-60a3ce03674c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e9153bf-6d19-4aca-b797-60a3ce03674c" (UID: "1e9153bf-6d19-4aca-b797-60a3ce03674c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 01 17:42:23.104965 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.104948    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e9153bf-6d19-4aca-b797-60a3ce03674c" (UID: "1e9153bf-6d19-4aca-b797-60a3ce03674c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 01 17:42:23.106689 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.106664    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e9153bf-6d19-4aca-b797-60a3ce03674c" (UID: "1e9153bf-6d19-4aca-b797-60a3ce03674c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 01 17:42:23.106829 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.106806    2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-api-access-f25dn" (OuterVolumeSpecName: "kube-api-access-f25dn") pod "1e9153bf-6d19-4aca-b797-60a3ce03674c" (UID: "1e9153bf-6d19-4aca-b797-60a3ce03674c"). InnerVolumeSpecName "kube-api-access-f25dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 01 17:42:23.205350 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.205312    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e9153bf-6d19-4aca-b797-60a3ce03674c-proxy-tls\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:42:23.205350 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.205348    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:42:23.205577 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.205361    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e9153bf-6d19-4aca-b797-60a3ce03674c-kserve-provision-location\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:42:23.205577 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.205375    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f25dn\" (UniqueName: \"kubernetes.io/projected/1e9153bf-6d19-4aca-b797-60a3ce03674c-kube-api-access-f25dn\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:42:23.205577 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.205388    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e9153bf-6d19-4aca-b797-60a3ce03674c-cabundle-cert\") on node \"ip-10-0-129-95.ec2.internal\" DevicePath \"\""
Jul 01 17:42:23.242253 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.242219    2576 generic.go:356] "Generic (PLEG): container finished" podID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerID="2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5" exitCode=0
Jul 01 17:42:23.242392 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.242275    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"
Jul 01 17:42:23.242392 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.242265    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerDied","Data":"2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5"}
Jul 01 17:42:23.242481 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.242395    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd" event={"ID":"1e9153bf-6d19-4aca-b797-60a3ce03674c","Type":"ContainerDied","Data":"72b6ffb99e6b1a9a632428d8f4db452929385eb8c77c9a65dc2fb936fc738aed"}
Jul 01 17:42:23.242481 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.242424    2576 scope.go:117] "RemoveContainer" containerID="2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818"
Jul 01 17:42:23.251108 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.251084    2576 scope.go:117] "RemoveContainer" containerID="2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5"
Jul 01 17:42:23.258112 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.258095    2576 scope.go:117] "RemoveContainer" containerID="a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0"
Jul 01 17:42:23.264577 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.264554    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"]
Jul 01 17:42:23.265041 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.265024    2576 scope.go:117] "RemoveContainer" containerID="2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818"
Jul 01 17:42:23.265279 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:42:23.265260    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818\": container with ID starting with 2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818 not found: ID does not exist" containerID="2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818"
Jul 01 17:42:23.265334 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.265288    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818"} err="failed to get container status \"2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818\": rpc error: code = NotFound desc = could not find container \"2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818\": container with ID starting with 2414992aa7afd86d1f1528d6412930266485ac04908c3d44fb409272b76cf818 not found: ID does not exist"
Jul 01 17:42:23.265334 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.265305    2576 scope.go:117] "RemoveContainer" containerID="2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5"
Jul 01 17:42:23.265562 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:42:23.265543    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5\": container with ID starting with 2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5 not found: ID does not exist" containerID="2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5"
Jul 01 17:42:23.265638 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.265571    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5"} err="failed to get container status \"2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5\": rpc error: code = NotFound desc = could not find container \"2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5\": container with ID starting with 2d5bcc318d01cbc100c64e444473bb9396e1aabdb05b6f18e8eb1bb73dfd58a5 not found: ID does not exist"
Jul 01 17:42:23.265638 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.265592    2576 scope.go:117] "RemoveContainer" containerID="a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0"
Jul 01 17:42:23.265821 ip-10-0-129-95 kubenswrapper[2576]: E0701 17:42:23.265805    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0\": container with ID starting with a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0 not found: ID does not exist" containerID="a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0"
Jul 01 17:42:23.265860 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.265826    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0"} err="failed to get container status \"a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0\": rpc error: code = NotFound desc = could not find container \"a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0\": container with ID starting with a03d867070c4adc0d66dc42f05462440032da3baee87b03f9961a6ab4530e9d0 not found: ID does not exist"
Jul 01 17:42:23.268545 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:23.268509    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-bc58e-predictor-db56bb776-xzgmd"]
Jul 01 17:42:24.047281 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:24.047246    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" path="/var/lib/kubelet/pods/1e9153bf-6d19-4aca-b797-60a3ce03674c/volumes"
Jul 01 17:42:57.632050 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:57.632024    2576 ???:1] "http: TLS handshake error from 10.0.137.141:49590: EOF"
Jul 01 17:42:57.635861 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:57.635837    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-psbvf_ff5e36dc-a802-4ffd-99ec-db7ceb2429ba/global-pull-secret-syncer/0.log"
Jul 01 17:42:57.724470 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:57.724444    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-88pqd_97908b6c-1862-4784-bb3a-463d7a4cc872/konnectivity-agent/0.log"
Jul 01 17:42:57.777119 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:42:57.777092    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-95.ec2.internal_a2c3d4ff9542796e61a3b334a5839879/haproxy/0.log"
Jul 01 17:43:02.619389 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:02.619310    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-gzrvs_b1c856c3-297e-45fb-acfc-676730877495/kube-state-metrics/0.log"
Jul 01 17:43:02.644922 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:02.644891    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-gzrvs_b1c856c3-297e-45fb-acfc-676730877495/kube-rbac-proxy-main/0.log"
Jul 01 17:43:02.668498 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:02.668467    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-845b954455-gzrvs_b1c856c3-297e-45fb-acfc-676730877495/kube-rbac-proxy-self/0.log"
Jul 01 17:43:02.973849 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:02.973821    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xbz58_64a2befe-28f1-4bbb-9d47-1ff5d1088269/node-exporter/0.log"
Jul 01 17:43:02.997139 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:02.997112    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xbz58_64a2befe-28f1-4bbb-9d47-1ff5d1088269/kube-rbac-proxy/0.log"
Jul 01 17:43:03.020304 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:03.020279    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xbz58_64a2befe-28f1-4bbb-9d47-1ff5d1088269/init-textfile/0.log"
Jul 01 17:43:03.424295 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:03.424207    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-nvcrd_6edd6a76-0086-4102-872a-62e611e07a98/prometheus-operator/0.log"
Jul 01 17:43:03.452846 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:03.452822    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7b7d896497-nvcrd_6edd6a76-0086-4102-872a-62e611e07a98/kube-rbac-proxy/0.log"
Jul 01 17:43:05.917313 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:05.917278    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5499b6fb8f-skzxj_38413f5e-8ad5-4def-9fed-a74c356789d7/download-server/0.log"
Jul 01 17:43:06.092537 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092470    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"]
Jul 01 17:43:06.092776 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092762    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="storage-initializer"
Jul 01 17:43:06.092835 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092779    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="storage-initializer"
Jul 01 17:43:06.092835 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092806    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container"
Jul 01 17:43:06.092835 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092813    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container"
Jul 01 17:43:06.092835 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092825    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kube-rbac-proxy"
Jul 01 17:43:06.092835 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092830    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kube-rbac-proxy"
Jul 01 17:43:06.092989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092894    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kserve-container"
Jul 01 17:43:06.092989 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.092901    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e9153bf-6d19-4aca-b797-60a3ce03674c" containerName="kube-rbac-proxy"
Jul 01 17:43:06.095967 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.095947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.098774 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.098752    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mf9xf\"/\"openshift-service-ca.crt\""
Jul 01 17:43:06.100085 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.100070    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mf9xf\"/\"default-dockercfg-vm2np\""
Jul 01 17:43:06.100169 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.100146    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mf9xf\"/\"kube-root-ca.crt\""
Jul 01 17:43:06.105930 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.105911    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"]
Jul 01 17:43:06.232301 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.232268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-lib-modules\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.232475 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.232314    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-proc\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.232475 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.232339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-sys\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.232475 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.232384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-podres\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.232475 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.232420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxkgn\" (UniqueName: \"kubernetes.io/projected/a443ed56-340a-4892-a89c-4337e1c13d0c-kube-api-access-kxkgn\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333393 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-lib-modules\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-proc\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-sys\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-podres\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxkgn\" (UniqueName: \"kubernetes.io/projected/a443ed56-340a-4892-a89c-4337e1c13d0c-kube-api-access-kxkgn\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333582    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-sys\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333541    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-lib-modules\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333557    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-proc\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.333616 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.333586    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a443ed56-340a-4892-a89c-4337e1c13d0c-podres\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.342168 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.342146    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxkgn\" (UniqueName: \"kubernetes.io/projected/a443ed56-340a-4892-a89c-4337e1c13d0c-kube-api-access-kxkgn\") pod \"perf-node-gather-daemonset-fjkqv\" (UID: \"a443ed56-340a-4892-a89c-4337e1c13d0c\") " pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.406586 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.406548    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:06.533561 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:06.533466    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"]
Jul 01 17:43:06.535637 ip-10-0-129-95 kubenswrapper[2576]: W0701 17:43:06.535608    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda443ed56_340a_4892_a89c_4337e1c13d0c.slice/crio-9353a45032bd9096dd154b042a11899850649566671aad7bfe7691931639dfe4 WatchSource:0}: Error finding container 9353a45032bd9096dd154b042a11899850649566671aad7bfe7691931639dfe4: Status 404 returned error can't find the container with id 9353a45032bd9096dd154b042a11899850649566671aad7bfe7691931639dfe4
Jul 01 17:43:07.406750 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:07.406716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv" event={"ID":"a443ed56-340a-4892-a89c-4337e1c13d0c","Type":"ContainerStarted","Data":"e1482be9a38666214e76819eaee1d4c0ffae4a1864a5314bb403e0d12c447a5e"}
Jul 01 17:43:07.407162 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:07.406775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv" event={"ID":"a443ed56-340a-4892-a89c-4337e1c13d0c","Type":"ContainerStarted","Data":"9353a45032bd9096dd154b042a11899850649566671aad7bfe7691931639dfe4"}
Jul 01 17:43:07.407162 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:07.406814    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:07.427494 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:07.427440    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv" podStartSLOduration=1.427419327 podStartE2EDuration="1.427419327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-01 17:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-01 17:43:07.425316058 +0000 UTC m=+1617.830976835" watchObservedRunningTime="2026-07-01 17:43:07.427419327 +0000 UTC m=+1617.833080104"
Jul 01 17:43:07.427760 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:07.427745    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kgg9j_6bce2dd3-45c1-4fbe-b853-3c97b6508100/dns/0.log"
Jul 01 17:43:07.459541 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:07.459499    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kgg9j_6bce2dd3-45c1-4fbe-b853-3c97b6508100/kube-rbac-proxy/0.log"
Jul 01 17:43:07.627238 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:07.627206    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lm9pd_c0ba3c39-1752-4fac-b0d3-8868786e2412/dns-node-resolver/0.log"
Jul 01 17:43:08.246882 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:08.246854    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jcctl_7373fb09-974e-47d2-8064-e91523f0f494/node-ca/0.log"
Jul 01 17:43:09.461668 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:09.461641    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nvbqg_18279d9a-bdc9-48f6-bb38-dfe5711b3460/serve-healthcheck-canary/0.log"
Jul 01 17:43:10.095262 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:10.095227    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhkql_0cf27d80-b131-4905-8ec0-6a15ea46fc8e/kube-rbac-proxy/0.log"
Jul 01 17:43:10.115543 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:10.115502    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhkql_0cf27d80-b131-4905-8ec0-6a15ea46fc8e/exporter/0.log"
Jul 01 17:43:10.136167 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:10.136144    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhkql_0cf27d80-b131-4905-8ec0-6a15ea46fc8e/extractor/0.log"
Jul 01 17:43:12.228508 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:12.228478    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-228h5_b6d29a38-1413-448e-b3c8-7c6c94feedf4/server/0.log"
Jul 01 17:43:12.308673 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:12.308638    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-6xmcq_dd2c74f9-bb34-4173-a7cd-826e7e8bf8a0/manager/0.log"
Jul 01 17:43:13.419558 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:13.419529    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mf9xf/perf-node-gather-daemonset-fjkqv"
Jul 01 17:43:18.262508 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.262482    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nh6jx_7e3f55f5-2715-4960-8d8f-546f4a38d95b/kube-multus-additional-cni-plugins/0.log"
Jul 01 17:43:18.286227 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.286199    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nh6jx_7e3f55f5-2715-4960-8d8f-546f4a38d95b/egress-router-binary-copy/0.log"
Jul 01 17:43:18.306697 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.306668    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nh6jx_7e3f55f5-2715-4960-8d8f-546f4a38d95b/cni-plugins/0.log"
Jul 01 17:43:18.329960 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.329909    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nh6jx_7e3f55f5-2715-4960-8d8f-546f4a38d95b/bond-cni-plugin/0.log"
Jul 01 17:43:18.352878 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.352849    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nh6jx_7e3f55f5-2715-4960-8d8f-546f4a38d95b/routeoverride-cni/0.log"
Jul 01 17:43:18.375238 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.375212    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nh6jx_7e3f55f5-2715-4960-8d8f-546f4a38d95b/whereabouts-cni-bincopy/0.log"
Jul 01 17:43:18.396144 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.396116    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nh6jx_7e3f55f5-2715-4960-8d8f-546f4a38d95b/whereabouts-cni/0.log"
Jul 01 17:43:18.645394 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.645317    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vczhz_f72fd9ff-c4f4-44d7-afb1-c7c8313e08b9/kube-multus/0.log"
Jul 01 17:43:18.716272 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.716242    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tb5ht_c7ba0dbe-db80-4c86-afc4-545af38d30d0/network-metrics-daemon/0.log"
Jul 01 17:43:18.737329 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:18.737297    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tb5ht_c7ba0dbe-db80-4c86-afc4-545af38d30d0/kube-rbac-proxy/0.log"
Jul 01 17:43:20.062429 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.062399    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-controller/0.log"
Jul 01 17:43:20.083125 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.083099    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/0.log"
Jul 01 17:43:20.083125 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.083102    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovn-acl-logging/1.log"
Jul 01 17:43:20.101916 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.101890    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/kube-rbac-proxy-node/0.log"
Jul 01 17:43:20.124415 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.124389    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 01 17:43:20.144045 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.144020    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/northd/0.log"
Jul 01 17:43:20.166736 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.166711    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/nbdb/0.log"
Jul 01 17:43:20.187816 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.187796    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/sbdb/0.log"
Jul 01 17:43:20.235677 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:20.235645    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtzvn_c12fa6c4-e35f-4aa2-8b8e-f274469878a9/ovnkube-controller/0.log"
Jul 01 17:43:21.335383 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:21.335350    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-nqm5t_a6f9ca68-b7ed-4b9e-a028-62e8c7d19a59/network-check-target-container/0.log"
Jul 01 17:43:22.444621 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:22.444591    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-l9hwb_e17c5967-c89a-47b9-9d37-3281f0f2b99c/iptables-alerter/0.log"
Jul 01 17:43:23.297448 ip-10-0-129-95 kubenswrapper[2576]: I0701 17:43:23.297419    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-5v4vn_5a686c1c-ce50-4e13-82d7-2296f4797d30/tuned/0.log"